What Comes After Google?

October 4, 2007

Question: What Comes After Google?

Yahoo just released a new Search Assistant feature this week (TechCrunch) (Read/WriteWeb). Ask has been trying a new interface lift for a while (TechCrunch) (Read/WriteWeb). While these are all very nice incremental improvements to search, are they enough to supplant Google? Do they tackle the fundamental problem of information retrieval in a paradigm shifting way? The answer is probably not.

Now imagine several years into the future. Will you find information in the same way in the future as you do today? Again, probably not.

This may sound like an obvious “duh”-ism, but its ramification certainly is not. As unfathomable as it may seem, Google, as we know it today, will probably not be how we find most of our information in a few years. Since Youlicit is an information retrieval company, we had to ask ourselves, “If not Google, then what?”

What is the logic that dictates the evolution of information retrieval paradigms?

Evolution of Information Retrieval Paradigms

To answer this question, we first plotted the different paradigms of information retrieval on a timeline. If we can figure out what the axis of this graph represents, then we should be able to predict which new solutions will succeed and which will not by simply identifying the solutions that maximize the metric along this axis.

If you’re a start-up, this understanding can guide you in building a successful innovative product. If you’re a venture capitalist or technology evaluator, this insight gives you a criterion for determining which technologies to invest in and which ones will fade away as fads of the day.

Evolution of Information Retrieval Paradigms

After plotting them on a timeline, we then explored the three major paradigms of information retrieval:

  1. Manual Organization
  2. Algorithmic Search
  3. User-Generated Recommendations

Manual Organization

Information retrieval, during its infancy, started off as a very rigid and structured process. Those who remember Gopher or Jerry and David’s Guide to the World Wide Web (later known as Yahoo) know how attempts were made to organize sites into a pre-determined hierarchy. However, as the number of web sites exploded exponentially, manually organizing sites into structured directories became practically impossible:

“A universal ontology is difficult and expensive to construct and maintain when there involve hundreds of millions of users with diverse background. When used to organize Web objects, ontology faces two hard problems: unlike physical objects, digital content is seldom semantically pure to fit in a specific category; and it is difficult to predict the paths, through which a user would explore to discover a digital object.”
Clay Shirkey

Algorithmic Search

Too many sites to categorize? No problem. Algorithmic search to the rescue. Web search engines, such as Altavista and Google, arrived and allowed the web to grow in its chaotic unstructured way while still providing a level of organization in the form of keywords. Now instead of having to know the correct directory hierarchy, users only needed to know the keyword combination (and page number) for sites they were looking for.

Counter to intuition, search engines actually decrease the relevance of individual results as compared to those in a manually organized directory. A hand-picked set of results are always better than an algorithmically generated set of results. However, since search engines have a much greater coverage of the Web, the average relevance of search results from a given set of topics is usually better than the average relevance of directory results on the same set of topics.

The other improvement made by search was the replacement of directory hierarchies by keywords as the primary recall mechanism. While still not perfect, guessing and checking keywords took a lot less effort than guessing and checking hierarchies. Seach engines effectively decreased the recall effort.

User-Generated Recommendations

Recently, we’ve witnessed the niche adoption of tagging, voting, stumbling and other “user-generated relevance” as a means of finding information. Why? It’s because they improved something along either the average relevance dimension or the recall effort dimension.

Take Del.icio.us and Digg for instance. In the scope of technology related content, the average relevance of results from these folksonomy sites is better than from search engines because these folksonomy sites have been able to increase coverage by effectively crowdsourcing an easier manual organization process.

StumbleUpon went the other route. Instead of improving average relevance, it decided to reduce the recall effort from guessing and checking keywords to a one-click no-thought “stumble.” In doing so, it did something ingenious: StumbleUpon removed the world’s most scarce resource from the information retrieval process… human thought.

Answer: The Solutions that Maximizes the “Search Metric”

As well as they’ve done, both kinds of user-generated recommendation services are plateauing well before crossing the chasm into the mainstream market. Why? We think it’s because they’ve only focused on a singular improvement, either average relevance or recall effort, but not both.

In order for an information retrieval solution to penetrate into the mass market, the solution has to take a dual approach. It has to concurrently maximize average relevance and minimize recall effort. It’s simply a matter of optimizing the Average Relevance / Recall Effort ratio, or as we like to call it, the “Search Metric.” The solution that does this best will probably gain the most mindshare and supplant algorithmic search as the primary mode of information retrieval. And that is what comes after Google.

Afterthought

Does this imply that algorithmic search will become extinct anytime soon? Absolutely not. It just means that more and more people will find larger percentages of their daily information through means other than search. Our bets are on online “word of mouth” or user-generated recommendations.

Evolution of Information Retrieval Paradigms

We’ve builit Youlicit with this assumption at the core. Youlicit is a “word of mouth” or recommendation engine (as opposed to a search engine). We’re trying to maximize the Search Metric by combining user-generated relevance with one-click no-thought recall. We want to improve the information retrieval landscape by enabling the average user to harness the wisdom of the crowds with very little effort. If you’re as obsessed with this problem as we are, we’d love nothing more than to hear from you!

— The Youlicit Team


Search vs. Recommendations

September 12, 2007

In an interesting blog entry by Seth Godin, he states:

“The fact is that search engines are very good at fairly simple searches, and very good at finding information about single products, services, people and ideas. But they’re terrible at connections, at rankings, at horizontal results… They can’t help me find six products that are viable alternatives to something that was just discontinued.”

These are precisely the problems we are trying to address at Youlicit. Our primary focus is drawing connections between horizontal results, or to put it more plainly providing “recommendations” on topics, through user provided data.

While Google and Yahoo! are “search” engines, Youlicit is a “recommendation” engine.

What’s the difference between search and recommendations you ask? Search is when you precisely know what you are looking for, whereas recommendations are when you aren’t entirely sure and would like some guidance on where to go and what to see. A great example is visiting your local department store. Search is the equivalent of going to the store and saying, “I’m looking for a navy blue dress shirt with thin vertical stripes.” Recommendations are when you walk to the mens department, pick up a blue shirt and say, “Can you show me more like this?”

As applied to web content, when you can accurately guess the keywords that are likely to occur on the pages you’re looking for, use search. But when you’re not quite sure how to describe what you’re looking, but you know it’s related to what you’re currently looking at, that’s when you Youlicit More recommmendations!