In section 6 they report the results of a survey where stakeholders were asked to rate the importance of the different TREC tracks. The most important tracks were Adhoc Track with 77% rating it very important and the Web Track with 74%. Other highly rated tracks were the TrecVid and Q&A tracks. In contrast, the Spam Track (mainly email) and Speech track were ranked at the bottom.
Here are few highlights from the conclusion. In monetary terms:
As described in Section 6, $16 million of discounted investments have made by NIST and others in TREC have resulted in $81 million in discounted extrapolated benefits or a net present value of $65 million.The report continues to say that the same rate of benefit may not continue because much of the growth happened in the late 90s with the rapid expansion of the PC and the use of the Web.
The other main section I found interesting was the possible future directions:
Several trends in survey responses emerged; 37 respondents indicated that TREC should expand into new tracks, 20 said TREC should develop new evaluation methods, and 17 said TREC should develop new data sets. Common suggestions were the following:
More broadly, several respondents suggested that TREC should work with industry to increase their participation in the TREC workshops, as well as to solicit data that they might allow the TREC audience to use, thus increasing the usefulness of TREC results...
- Focus on more user behavior data (e.g., social data, Twitter, geographically based) to improve on the Interactive track.
- Continue to look at multimedia search techniques (e.g., pictures, video).
- Expand into more focused search areas (e.g., chemistry, drug design, evidence-based medicine).