Monday, November 1

Blekko Launches: Brings Transparency to Relevance Ranking

blekko launched its public beta on Monday. blekko is a new web search engine that focuses on creating an open and transparent process around search engine relevance ranking. blekko is attempting to differentiate itself as the open alternative to "closed" search engines and involve greater public participation in the ranking process.

Creating blekko is an impressive feat because they built their own system from the ground up to crawl, index, and rank a multi-billion page search index. This is hard to do well. They have accomplished a lot in short period of time, so I am excited about the changes we'll see as they evolve. I hope that they will take the risks that other search engines can't afford. One of their risky moves is opening up their ranking features.

Open vs. Closed Ranking
Google's "closed policy" is a difficult issue that has garnered significant criticism. For example, at ECIR 2008 in a QA with Amit Singhal, Daniel Tunkelang questioned the need to rely on security through obscurity. (For an updated perspective now that Daniel works at Google read his recent post on Google and Transparency). In response to a EU inquiry, Amit Singhal laid out the underlying philosophy of Google's ranking,
  1. Algorithmically-generated results.
  2. No query left behind.
  3. Keep it simple.
Although Google uses signals from humans in the form of links and click through on search results, it does not actively involve humans in the search process. Blekko is going to be different.

As a first step involving users in ranking, blekko allows users define their own search experience using "slashtags". Founder Rich Skrenta describes this on a recent blog post, on crowdsourcing relevance,
We're starting by letting users define their own vertical search experiences, using a feature we call slashtags. Slashtags let all of the vertical engines that people define on blekko live within the same search box. They also let you do a search and quickly pivot from one vertical to another.
You can contrast Google's philosophy with Blekko's, here are the first 3 of 11 points in Rich Skrenta's post,
  1. Search shall be open
  2. Search results shall involve people
  3. Ranking data shall not be kept secret
  4. ...
A philosophy is great, but it doesn't matter if your results suck. Blekko just launched, so let's take a closer look.

Blekko's ranking
I tried blekko and it is a very solid first effort. To experiment, I re-ran a variety of searches from my Google web history. I didn't conduct thorough experiments, but my impression is that the ranking and coverage is very reasonable, but not as good as Google or Bing's. SELand has a more comprehensive review with a side-by-side comparison with Google.

One frustration I encountered using blekko is that slashtags autofired and my query was automatically restricted to a vertical when it was overly restricted. This limited scope led to missing key relevant results and I manually backed off several times to the /web. Slashtags create added complexity which leads to problems.

I'd like to point out a few queries where I found that blekko's relevance particularly stumbled and could improve, [carbonell mmr] and [iron cook northampton]. Neither of these are easy queries. The first is somewhat ambiguous and the second is about a small local event. What I find hopeful with blekko is that I can begin to understand the underlying reason for failure. I clicked on "rank stats" or you can use the /rank tag, e.g. [carbonell mmr /rank]. For each result blekko also provides an "seo" link to see static rank features, http://www.ml.cmu.edu/mlpeople/affiliatedfaculty.html. As an IR researcher I find open access to this feature data very exciting. However, for the average searcher this level of detail is distracting and unnecessary. The "openness" features need to earn their real estate by being actionable, but right now they don't do that.

Instead of cluttering the search UI, I would like to see blekko be more open by providing the data through an API. It would let academics and searchers use this raw material to rerank results in new and novel ways.

On "Crowdsourcing relevance"
Slashtags are operators that can both restrict a search and change its ranking. They currently allow you to sort the results by /relevance or /date. Users can define slashes that tag hosts as relevant to topic. I started a slashtag on information-retrieval. However, restricting a query to a set of hosts using slashtags is a bit like performing surgery with a chainsaw. In the end you are missing key bits. This approach has several problems:
  1. The granularity of hosts is too coarse. The amount of content relevant to a topic could be a single page or section of website.

  2. Recall. A slashtag cannot be mantained by people in real-time and will miss relevant content.

  3. The semantics of a slashtag are not well defined and it is not obvious how to combine them, e.g. a combine a topic slashtag with a /date ranking.
The claim is that slashtags reduce spam by limiting search to a restricted set of trusted websites. However, I don't think that the impact of spam on search results is very compelling. Search engines are quite good at identifying and incorporating implicit user feedback to reduce the impact of irrelevant (spammy) results. There needs to be a more compelling reason.

Using slashtags doesn't address several key issues in crowdsourcing ranking. The first is that they doesn't address the obvious need to involve people in making relevance assessments on the results in a systematic way. Secondly, the core of search ranking is determining what features indicate relevance for a query and how they should be combined. blekko is not currently surfacing a way to change either of those aspects.

It remains to be seen how you could really let users change the relevance in meaningful ways and more importantly, measure the utility to everyone. It may be that academia could play some role in creating and testing features.

Presentation
blekko's "10 blue links" search UI feels outdated. Modern search engines are incorporating rich results into SERPs. The "Universal Search" results blend images, videos, maps, even places and people. I hope that this is an area where we see blekko evolve quickly to catch up.

Overall
I won't be switching to blekko for regular use. I find the level of ranking information and features that they share very exciting and compelling. Because I can see the ranking pieces, it compels me to jump in and help make things better, because I can. However, I question the utility of the information for average users and the ability to deeply engage the public in useful ways to improve ranking.

Slashtags are fun to play with, but are they useful? Slicing the web into groups creates mini vertical search experiences. However, using the tags adds complexity that may not be necessary most of the time. The value offered by the slashtag verticals is quite limited right now. I hope that slashtags will evolve to allow users to do more curation and add more value as blekko matures.

3 comments:

  1. Great post! I posted my own reaction to Blekko during the beta: Taking Blekko out for a Spin. I'd really like to see the sites used to specify slashtags as seeds to train classifiers that could then be applied to the entire index. Would that be enough to make Blekko successful? I don't know, but it would certainly make their most visible differentiator a lot more compelling.

    ReplyDelete
  2. crowd sourcing the queries is already hard enough, not to mention the search results.
    see http://boston.lti.cs.cmu.edu/wikiquery/

    ReplyDelete
  3. Anonymous9:18 PM EST

    Good catch, thanks Jeff! I also find this SE (especially bits and pieces about ranking) very exciting from an IR researcher point-of-view.. On the other hand, I am not sure how the paradigm differs from delicious or even DMOZ? I am not sure whether delicious has "search under tag" option, but the idea of slashtags seems like to do so... Still, interesting...

    ReplyDelete