The Picture This game randomly pairs two users in a game setting. The game selects queries, then the players are shown two or more images. They then decide which image is most relevant to the query.If you're just getting started with preference data, you should check out Ben Carterette's recent publications. As part of his thesis he created a preference test collection for data from the TREC 2003 web track.
Tuesday, March 10
Live Labs Releases Image Preference Data
Jon Elsas highlighted Live Labs' release of an image preference dataset. They have a paper describing their research in an upcoming WWW paper, Learning Consensus Opinion: Mining Data from a Labeling Game. The preference dataset contains 427 queries. The data was collected using an ESP-like game,