Tim O'Reilly wrote an article, Search Startups are Dead, Long Live Search Startups. His article itself is a response to Tim Burnham's article of the same title.
I really enjoyed reading the interesting discussion from on Tim's blog. It garnered lots of interesting discussion. Highlights include Tim's comments along with those of Ken Krugler (Krugle.com), Otis Gospodnetic (Simpy), Markl (Google Custom search engine), and Jeff Chan (TopicBlogs).
To summarize Burnham's original article:
The hardware and bandwidth needed to create a large web scale search engine (high performance crawling, indexing, query serving, link analysis, etc...) provides a significant barrier to entry for new startups. In short, startups are effectively priced out of the market and that VCs and entrepeneurs should think hard before (don't even bother) attempting to displace one of GYM.
His message is that it won't be long before web search is a commodity platform. This is signaled by fledgling services such as Alexa and Google Custom Search Engine. The idea is that startups could switch search providers as easily as they could switch between RDBMs today. In other words, developers should focus on developing innovative applications that uses these new web search services rather than the attempting to write SQL Server from scratch.
Tim agrees and writes in his comments:
I'm not suggesting that there isn't the possible of radical disruption to the current model. But I still believe that we're entering the centralization phase of the web, in which the big get bigger, and put up barriers to entry to the new guys.One of the best comments was from Jeff Chan who astutely writes:
I agree with Tim and Jeff. You can't create just another search engine and expect to succeed. There will need to be a radical disruption in order to displace Google. In my opinion, search is not yet a commodity platform; there are to few vendors and their services are unproven and immature. I wouldn't bet my (search engine) business on them, yet.
It appears to be trendy to point to infrastructure as a barrier to entry for almost any new (or existing) category of services. A few hundred servers suffice for a small number of copies of a web-scale crawl and index. Google's scale is necessary for the volume of queries it serves, not the size of the corpus.
The real problems potential entrants have to deal with are technology and distribution. It is an open question whether anyone can significantly improve on the existing approach (perhaps mining and summarizing facts?), and even if they could, whether they can acquire a following before an existing service replicates it or acquires them.