tag:blogger.com,1999:blog-18315968.post6830386365366163832..comments2008-06-17T08:48:11.926-04:00jeff.daltonhttp://www.blogger.com/profile/12887721174386884522noreply@blogger.comBlogger2125tag:blogger.com,1999:blog-18315968.post-7294143574538365332008-06-17T08:48:00.000-04:002008-06-17T08:48:00.000-04:00Thanks for the pointer to the test.<BR/><BR/>An interesting start, but I have more than a few issues with the way they did their test. My biggest issue is with their evaluation metric. <BR/><BR/>"For each engine, we count the number of queries that had at least one "Highly Relevant" result within the first five results the engine returned. This is a version of the "precision at 5" metric from information retrieval."<BR/><BR/>They aren't using P@5, see the article on <A HREF="http://en.wikipedia.org/wiki/Information_retrieval#Performance_measures" REL="nofollow">IR evaluation metrics</A>. P@5 is relevant retrieved / retrieved. <BR/><BR/>Basically, the metric they are using isn't meaningful. A better metric would be normalized discounted cumulative gain (NDCG)@10. Perhaps even better, use pairwise preference judgments: <A HREF="http://ciir.cs.umass.edu/~carteret/ecir08.pdf" REL="nofollow">Here or There: Preference Judgments for Relevance</A>jeff.daltonhttp://www.blogger.com/profile/12887721174386884522noreply@blogger.comtag:blogger.com,1999:blog-18315968.post-9875052925649795632008-06-16T05:05:00.000-04:002008-06-16T05:05:00.000-04:00At Dolores Lab they have done an empirical evaluation of the 4 major search engines and found no statistical difference between Google, Yahoo and Live. Only Ask performs worst.<BR/><BR/>They used Amazon's Mechanical Turk to conduct the tests. I was surprised with the tie in the top three.<BR/><BR/>http://blog.doloreslabs.com/2008/04/search-engine-relevance-an-empirical-test/Sérgio Nuneshttp://sergionunes.comnoreply@blogger.com