Greg Linden on evaluating recommendation systems like search

Greg Linden, creator of the original Amazon recommendation system, wrote an article on the new Communications of the ACM (CACM) blog on the difficulty evaluating recommendation systems.

Specifically, he picks on root-mean squared error (RMSE), the metric being used to evaluate the Netflix prize. He parallels it with web search, where you really want to find the top N best movies for you:

Web search engines primarily care about precision (relevant results in the top 10 or top 3). They only care about recall when someone would notice something they need missing from the results they are likely to see. Search engines do not care about errors scoring arbitrary documents, just their ability to find the top N documents.

Greg would welcome your thoughts and comments, and the new CACM blog could use more subscribers.