Friday, July 16

The Impact of TREC and its Future Directions

This week NIST released a report on the Economic Impact Assessment of TREC.

In section 6 they report the results of a survey where stakeholders were asked to rate the importance of the different TREC tracks. The most important tracks were Adhoc Track with 77% rating it very important and the Web Track with 74%. Other highly rated tracks were the TrecVid and Q&A tracks. In contrast, the Spam Track (mainly email) and Speech track were ranked at the bottom.

Here are few highlights from the conclusion. In monetary terms:
As described in Section 6, $16 million of discounted investments have made by NIST and others in TREC have resulted in $81 million in discounted extrapolated benefits or a net present value of $65 million.
The report continues to say that the same rate of benefit may not continue because much of the growth happened in the late 90s with the rapid expansion of the PC and the use of the Web.

The other main section I found interesting was the possible future directions:
Several trends in survey responses emerged; 37 respondents indicated that TREC should expand into new tracks, 20 said TREC should develop new evaluation methods, and 17 said TREC should develop new data sets. Common suggestions were the following:
  • Focus on more user behavior data (e.g., social data, Twitter, geographically based) to improve on the Interactive track.
  • Continue to look at multimedia search techniques (e.g., pictures, video).
  • Expand into more focused search areas (e.g., chemistry, drug design, evidence-based medicine).
More broadly, several respondents suggested that TREC should work with industry to increase their participation in the TREC workshops, as well as to solicit data that they might allow the TREC audience to use, thus increasing the usefulness of TREC results...

1 comment:

  1. Jeff,

    >In contrast, the Spam Track (mainly email) and
    >Speech track were ranked at the bottom.

    If you only use the "very important" opinions, then you are wrong: Speech track is better ranked than Enterprise track for example.

    You should use the sum of both "somewhat important" and "very important" opinions. You cannot only compare on "very important" opinions. In fact, it makes more sense to rank tracks on the "Not very important for IR" (lowest score first).

    In fact, if you sum both "somewhat important" and "very important" opinions, then the Web track is ranked better than Adhoc track (which probably makes sense) - while 5% thought that the Adhoc track is not very important to IR research, 0% of respondents found Web track not to be important for IR research.