TL;DR
This paper summarizes the TREC Novelty Detection track, which aimed to identify sentences containing relevant and new information in a document stream. The key takeaway is that while systems struggled to match human performance in distinguishing truly novel content, especially for opinions, the TREC experience provided valuable insights and datasets for future research in passage retrieval and summarization.
What Problem Does This Paper Address?
Traditional information retrieval systems focus on finding documents relevant to a user's query. However, users often need more than just relevant documents; they need to be alerted to new information within those documents, especially when dealing with a continuous stream of information like news. This is the core of "novelty detection." The challenge lies in identifying not just what is relevant to a user's interest, but also what is new to them, given what they've already seen. This is analogous to a system highlighting the most important and fresh updates in a news feed or a research paper. The TREC Novelty track aimed to operationalize and evaluate systems for this task, specifically at the sentence level.
Methods
The TREC Novelty track involved participants developing systems to identify sentences that were both relevant to a given topic and novel, meaning they hadn't appeared in previously seen documents for that topic. The task was broken down into four sub-tasks, allowing for different levels of training data (relevance and novelty information) to be provided.
- Passage Retrieval: A fundamental component was identifying relevant sentences. This is a form of passage retrieval, where the system must consider the sentence within its surrounding context, unlike traditional document retrieval.
- Novelty Detection: Once relevant sentences were identified, the system had to determine if they contained new information not present in earlier documents.
- Evaluation Metric: Due to the set-based nature of the task (unranked sets of sentences), traditional metrics like Mean Average Precision were not suitable. Instead, the F-measure, a harmonic mean of precision and recall, was used. This metric is robust to variations in the size of the "ground truth" sets.
Datasets
The TREC Novelty track experiments were conducted from 2002 to 2004.
- 2002 Data: This initial year's data was found to be problematic due to assessor disagreement and confusing instructions, and is not used in the paper's analysis.
- 2003 & 2004 Data: For these years, assessors created their own topics and selected relevant documents. Crucially, they also identified relevant sentences and then marked which of those were novel.
- Topic Types: The data was divided into two types: "event" topics (e.g., bombings) and "opinion" topics (e.g., gun control, cloning). This distinction proved important for understanding system performance.
- Document Sets: Each topic was associated with an ordered set of sentence-segmented documents. For tasks 2-4, "sentence qrels" (ground truth for relevance and novelty) were provided as training data.
- Irrelevant Documents: In 2004, some irrelevant documents were included to increase the "noise" in the data.
Key Insights and Ideas
- Opinions are Harder than Events: Systems (and humans) found it significantly more difficult to identify novel opinions compared to novel events. While systems could identify relevant sentences for opinions, determining their novelty was a greater challenge.
- Human Selectivity vs. System Over-Retrieval: Human assessors were remarkably selective, identifying a very small percentage of sentences as relevant (median 2% in 2002, 19.2% in 2004). In contrast, systems tended to retrieve a much larger proportion of sentences as relevant (average 49.5% in 2004). This suggests systems often err on the side of too much retrieval.
- Consecutive Sentences: Human assessors tended to select small blocks of consecutive sentences, whereas systems often identified many individual, non-consecutive sentences.
- Novelty is Difficult to Operationalize: Even for humans, judging novelty is subjective and relies heavily on memory and organizational skills. This makes creating reliable ground truth data challenging.
- Statistical vs. Linguistic Approaches: While statistical methods (like tf.idf and Okapi) were common, some groups explored linguistic approaches (deep parsing, coreference resolution). The paper suggests that while linguistic methods might offer higher precision, they often sacrifice recall.
Key Results
- Human Performance: In 2004, human assessors marked less than 20% of sentences in relevant documents as relevant, and only about 8% as both relevant and novel.
- System Performance (2004 Task 1): Systems retrieved an average of 49.5% of all sentences as relevant, compared to the human assessor's 19.2%. Systems also declared 41% of all sentences as novel, versus the human's 8.4%.
- F-scores: F-scores for relevance and novelty retrieval were generally uniform across systems, often dominated by the precision component. Top-scoring systems were often statistical.
- Task 2 (Novelty given relevant sentences): Most systems in Task 2 (where all relevant sentences were provided) labeled nearly every sentence as novel, indicating a general difficulty in distinguishing true novelty. Systems that performed better were often recall-oriented.
- Topic Type Impact: In Task 1, systems generally performed better on event topics than opinion topics. However, in Task 2, this distinction was less pronounced, suggesting the difficulty for opinions lies more in identifying relevance than novelty.
Limitations and Future Work
- Limitations:
- The paper notes that the 2002 data was flawed and not used for analysis.
- The inherent subjectivity of novelty judgments makes creating perfect ground truth difficult.
- Systems struggled to make fine-grained distinctions at the sentence level, especially when documents as a whole were relevant.
- Identifying relevant opinion sentences was particularly challenging.
- Future Work:
- The TREC novelty data can be used for passage retrieval evaluation, testing consecutive sentence retrieval, and as ground truth for extractive summarization.
- Further research is needed to better understand user agreement issues and the nuances of novelty detection.
- The paper suggests that while systems are recall-oriented, achieving high precision without sacrificing too much recall remains a key challenge.
Summary updated