Computing Community Consortium Blog

The goal of the Computing Community Consortium (CCC) is to catalyze the computing research community to debate longer range, more audacious research challenges; to build consensus around research visions; to evolve the most promising visions toward clearly defined initiatives; and to work with the funding organizations to move challenges and visions toward funding initiatives. The purpose of this blog is to provide a more immediate, online mechanism for dissemination of visioning concepts and community discussion/debate about them.


NIST Seeking Submissions to Text REtrieval Conference

January 16th, 2012 / in big science, research horizons, resources / by Erwin Gianchandani

NIST Issues Call for Participation for its 2012 Text REtrieval Conference (TREC) (image courtesy NIST).The National Institutes of Standards and Technology (NIST) — which recently posted a solicitation containing opportunities for computing researchersis now out with a call for submissions to its 21st annual Text REtrieval Conference (TREC), “the premier experimental effort in the field to encourage research in information retrieval and related applications” by providing a large test collection, uniform scoring procedures, and a forum for organizations interested in comparing their results. TREC has the following goals:

  • to encourage research in information retrieval based on large test collections;
  • to increase communication among industry, academia, and government by creating an open forum for the exchange of research ideas;
  • to speed the transfer of technology from research labs into commercial products by demonstrating substantial improvements in retrieval methodologies on real-world problems; and
  • to increase the availability of appropriate evaluation techniques for use by industry and academia, including development of new evaluation techniques more applicable to current systems.

According to a NIST press release:

Finding valuable information rapidly is much more than a game for people with a high-tech phone. Text retrieval is a field of research that can save lives by helping medical researchers locate key patient information or aid lawyers seeking important data in large digital data collections — both modern-day examples of needles in haystacks…

 

TREC is a rather unusual conference in that it starts months ahead of the actual meeting (Nov. 6-9, 2012) with the distribution of test data sets and challenges that TREC participants will use to develop and test advanced text retrieval techniques…

 

Scientists from academia and public and private-sector organizations … focus on improving information retrieval in specific areas. The groups develop algorithms to find information from [the] large, challenging datasets … provided by NIST. They work throughout the year and come to NIST’s headquarters in Gaithersburg, MD, to discuss their findings at the November meeting.

 

TREC is overseen by a program committee consisting of representatives from government, industry, and academia. For each TREC, NIST provides a test set of documents and questions. Participants run their own retrieval systems on the data, and return to NIST a list of the retrieved top-ranked documents. NIST pools the individual results, judges the retrieved documents for correctness, and evaluates the results…

 

A recent economic impact study prepared for NIST found that the NIST-led TREC project has significantly improved the ability to retrieve digital data. The report notes that TREC-related improvements are responsible for about one-third of the web-search advances between 1999 and 2009 and that the improvements may have saved up to 3 billion hours of web-search time.

This year’s TREC features 8 tracks:

Contextual Suggestions Track — investigates search techniques for complex information needs that are highly dependent on context and user interests.

 

Crowdsourcing Track — investigates emerging crowd-based methods for search evaluation and/or developing hybrid automation+crowd search systems.

 

Knowledge Base Acceleration Track — looks to develop techniques to dramatically improve the efficiency of (human) knowledge base curators by having the system suggest modifications/extensions to the KB based on its monitoring of the data streams.

 

Legal Track —  to develop search technology that meets the needs of lawyers to engage in effective discovery in digital document collections.

 

Medical Records Track — to foster research on providing content-based access to the free-text fields of electronic medical records.

 

Microblog Track — examines search tasks and evaluation methodologies for information seeking behaviors in microblogging environments.

 

Session Track — to provide the necessary resources in the form of test collections to simulate user interaction and help evaluate the utility of an IR system over a sequence of queries and user interactions, rather than for a single “one-shot” query.

 

Web Track — explores Web-specific retrieval tasks, including diversity and efficiency tasks, over collections of up to one billion Web pages.

And here’s a timeline for those interested in participating this year:

As soon as possible — submit your application to participate in TREC 2012:

Submitting an application will add you to the active participants’ mailing list. On Feb 23, NIST will announce a new password for the “active participants” portion of the TREC web site.

 

Beginning March 1

Document disks used in some existing TREC collections distributed to participants who have returned the required forms. Please note that no disks will be shipped before March 1.

 

July–August:

Results submission deadline for most tracks. Specific deadlines for each track will be included in the track guidelines, which will be finalized in the spring.

 

September 30 (estimated):

Relevance judgments and individual evaluation scores due back to participants.

 

November 6-9:

TREC 2012 conference at NIST in Gaithersburg, MD.

To learn more, and to apply to participate, check out the TREC 2012 website.

(Contributed by Erwin Gianchandani, CCC Director)

NIST Seeking Submissions to Text REtrieval Conference

Comments are closed.