What Comes After Google?

Question: What Comes After Google?

Yahoo just released a new Search Assistant feature this week (TechCrunch) (Read/WriteWeb). Ask has been trying a new interface lift for a while (TechCrunch) (Read/WriteWeb). While these are all very nice incremental improvements to search, are they enough to supplant Google? Do they tackle the fundamental problem of information retrieval in a paradigm shifting way? The answer is probably not.

Now imagine several years into the future. Will you find information in the same way in the future as you do today? Again, probably not.

This may sound like an obvious “duh”-ism, but its ramification certainly is not. As unfathomable as it may seem, Google, as we know it today, will probably not be how we find most of our information in a few years. Since Youlicit is an information retrieval company, we had to ask ourselves, “If not Google, then what?”

What is the logic that dictates the evolution of information retrieval paradigms?

Evolution of Information Retrieval Paradigms

To answer this question, we first plotted the different paradigms of information retrieval on a timeline. If we can figure out what the axis of this graph represents, then we should be able to predict which new solutions will succeed and which will not by simply identifying the solutions that maximize the metric along this axis.

If you’re a start-up, this understanding can guide you in building a successful innovative product. If you’re a venture capitalist or technology evaluator, this insight gives you a criterion for determining which technologies to invest in and which ones will fade away as fads of the day.

Evolution of Information Retrieval Paradigms

After plotting them on a timeline, we then explored the three major paradigms of information retrieval:

  1. Manual Organization
  2. Algorithmic Search
  3. User-Generated Recommendations

Manual Organization

Information retrieval, during its infancy, started off as a very rigid and structured process. Those who remember Gopher or Jerry and David’s Guide to the World Wide Web (later known as Yahoo) know how attempts were made to organize sites into a pre-determined hierarchy. However, as the number of web sites exploded exponentially, manually organizing sites into structured directories became practically impossible:

“A universal ontology is difficult and expensive to construct and maintain when there involve hundreds of millions of users with diverse background. When used to organize Web objects, ontology faces two hard problems: unlike physical objects, digital content is seldom semantically pure to fit in a specific category; and it is difficult to predict the paths, through which a user would explore to discover a digital object.”
Clay Shirkey

Algorithmic Search

Too many sites to categorize? No problem. Algorithmic search to the rescue. Web search engines, such as Altavista and Google, arrived and allowed the web to grow in its chaotic unstructured way while still providing a level of organization in the form of keywords. Now instead of having to know the correct directory hierarchy, users only needed to know the keyword combination (and page number) for sites they were looking for.

Counter to intuition, search engines actually decrease the relevance of individual results as compared to those in a manually organized directory. A hand-picked set of results are always better than an algorithmically generated set of results. However, since search engines have a much greater coverage of the Web, the average relevance of search results from a given set of topics is usually better than the average relevance of directory results on the same set of topics.

The other improvement made by search was the replacement of directory hierarchies by keywords as the primary recall mechanism. While still not perfect, guessing and checking keywords took a lot less effort than guessing and checking hierarchies. Seach engines effectively decreased the recall effort.

User-Generated Recommendations

Recently, we’ve witnessed the niche adoption of tagging, voting, stumbling and other “user-generated relevance” as a means of finding information. Why? It’s because they improved something along either the average relevance dimension or the recall effort dimension.

Take Del.icio.us and Digg for instance. In the scope of technology related content, the average relevance of results from these folksonomy sites is better than from search engines because these folksonomy sites have been able to increase coverage by effectively crowdsourcing an easier manual organization process.

StumbleUpon went the other route. Instead of improving average relevance, it decided to reduce the recall effort from guessing and checking keywords to a one-click no-thought “stumble.” In doing so, it did something ingenious: StumbleUpon removed the world’s most scarce resource from the information retrieval process… human thought.

Answer: The Solutions that Maximizes the “Search Metric”

As well as they’ve done, both kinds of user-generated recommendation services are plateauing well before crossing the chasm into the mainstream market. Why? We think it’s because they’ve only focused on a singular improvement, either average relevance or recall effort, but not both.

In order for an information retrieval solution to penetrate into the mass market, the solution has to take a dual approach. It has to concurrently maximize average relevance and minimize recall effort. It’s simply a matter of optimizing the Average Relevance / Recall Effort ratio, or as we like to call it, the “Search Metric.” The solution that does this best will probably gain the most mindshare and supplant algorithmic search as the primary mode of information retrieval. And that is what comes after Google.

Afterthought

Does this imply that algorithmic search will become extinct anytime soon? Absolutely not. It just means that more and more people will find larger percentages of their daily information through means other than search. Our bets are on online “word of mouth” or user-generated recommendations.

Evolution of Information Retrieval Paradigms

We’ve builit Youlicit with this assumption at the core. Youlicit is a “word of mouth” or recommendation engine (as opposed to a search engine). We’re trying to maximize the Search Metric by combining user-generated relevance with one-click no-thought recall. We want to improve the information retrieval landscape by enabling the average user to harness the wisdom of the crowds with very little effort. If you’re as obsessed with this problem as we are, we’d love nothing more than to hear from you!

— The Youlicit Team

Advertisements

3 Responses to What Comes After Google?

  1. Pat says:

    What about an intelligent agent that can:
    1. understand the query using NLP
    2. ask clarifying questions and/or present sub-categories to follow
    3. use nlp on the web pages to understand the meaning
    4. summarize from numerous web sources to answer the question
    5. point the user to the source docs the summaries came from

    Admittedly we are a ways away from anyone demonstrating this but many are working on it.

    Pat

  2. Toufique says:

    Hi Pat,

    You’re absolutely right. NLP is a fascinating area of AI research that holds a lot of promise if applied correctly. I’ll dissect it using the same framework:

    Avg. Relevance: Using steps 3-5, NLP can certainly improve the avg relevance by understanding the actual “meaning” of a page, instead of just checking for the presence of a keyword.

    Recall Effort: I don’t see NLP vastly improving keyword query entry because keyword search has already conditioned us to enter as few keywords as possible to describe what we’re looking for. Users don’t want to type in “When is Able Lincoln’s birthday?” when “Abe Lincoln birthday” works just fine. Whenever we see a search box, grammar tends to go out the window. I think the true potential of NLP as a query mechanism will be realized after speech recognition matures. NLP has a huge opportunity to reduce recall effort for spoken queries. That should be very interesting.

    Toufique

  3. […] of a search tool is to help you find the information you need with the least effort possible (see Relevance/Effort metric). To this end, if there is someone who has, and is willing to share, the information or expertise […]

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: