The many faces of Search: Larry Page, Sergei Brin, Jimmy Wales, Danny Sullivan, Jason Calacanis, Robert Scoble, Rand Fishkin and baby Taptu

There’s a debate raging in the blogosphere on the merits of algorithmic search (e.g. Google) vs human-powered search (e.g. Mahalo). It started at the end of last year, when Jimmy Wales got plenty of airtime talking about Search Wikia – a different approach to human-powered search. More recently, Scoble has stoked up the debate and has taken a lot more flak for advocating human-powered search as the way of the future. So much so that he was forced to go mea culpa on some of his debating points. But to paraphrase Scoble, what if there really is 10% real fruit juice in his drink?

The constraints of algorithmic search show up most clearly in challenging environments like mobile search. It’s the most challenging environment for search today because of the small screens, severe bandwidth constraints and high latencies associated with mobile networks. Talented teams have been trying for more than five years to make algorithmic search which works well on the desktop work well on mobile. We are still at less than 1% of the usage volume of desktop search.

But human-powered search as advocated by Jason Calacanis is not the whole answer either. Responding to Rand Fishkins rebuttal of Scoble, Julie Magro sums it up in a nutshell:

“I think you are right about the scalability issue. For a general search like HDTV, the Mahalo results seem to be better. But how can they consistently produce these results for searches that cannot be easily categorized?

For example, someone I know was recently diagnosed with Sarcoidosis. I got the email, did a Google search and within minutes found out what it was, how to treat it, and where to research it even more by doing a Google search (all with a baby in the other hand). I just tried the same serach on Mahalo…no results from them, they had to give me the Google results. The reason people use Google is because it works consistently for them. They know where to go to find answers.”

But let’s not throw Scoble’s and Wales’ baby out with the bathwater. There is a third way in search – Social-Assisted Search - which can potentially combine the best of both worlds. What is Social-Assisted Search? A virtuous combination of Long Tail coverage as previously delivered only by the algorithmic engines, and social accuracy as expressed in human voting and editorial judgement. Hardly anyone is talking about it - when I search on the phrase in Google today it is actually a Googlewack - a web metrics guy named Marshall Sponder seems to have invented the phrase.

As of 2007 the Social Web gives Long Tail coverage in way that it didn’t a few years ago. Illustrating this, Wikipedia does pretty well on Julie’s Sarcoidosis example. Similarly, MySpace provides rather amazing Long Tail coverage of songs – we’ve counted 8.5 million of them as of September 2007.

Humans are good at classifying the Web into categories then locating and aggregating the encyclopaedic sources for those categories (by encyclopaedic I mean comprehensive, and not literally encyclopedia-like). Machines are good at crawling and indexing and ranking the content contained and listed by these sources – that’s what a search engine does. A Social-Assisted Search engine that combines these processes would give you the holy grail of very pure and relevant search results and pretty good Long Tail coverage. The price you would pay compared to algorithmic search would be a reduction in the variety of results. You might end up with an index of 1 billion pages rather than the 20 billion plus index of the full Web.

On the desktop this price is probably not worth paying. But on mobile, where the user gets punished much more heavily when they inspect candidate results (by having to perform many more clicks and scrolls because of the small screen size, and sitting waiting for the wireless network to respond) this price IS worth paying.

Over the last 18 months at Taptu we have found Social-Assisted Search to be the most fruitful line of enquiry by far in the quest to bring about a quantum improvement in the performance of mobile search engines Delivering on the promise is a big engineering effort, so it’s going to take a while before Social-Assisted Search can match or beat the best of the algorithmics. But it’s coming.

This entry was posted on Thursday, September 6th, 2007 at 3:31 pm and is filed under Search. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.