Abstract
We demonstrate the value of using context in a new-information detection system that achieved the highest precision scores at the Text Retrieval Conference's Novelty Track in 2004. In order to determine whether information within a sentence has been seen in material read previously, our system integrates information about the context of the sentence with novel words and named entities within the sentence, and uses a specialized learning algorithm to tune the system parameters.
Introduction
In an era of digital information overload, users tracking news events often encounter the same information repeated across multiple sources. Novelty detection aims to filter out the "old" and highlight only the "new." A major bottleneck in the field is the difficulty of achieving high precision—correctly identifying only truly new information—while maintaining acceptable recall. Most existing systems at the time relied on sentence-by-sentence comparisons or simple word-overlap metrics, which fail to account for how human language flows. For instance, a sentence might contain zero "new" words but still be novel because it uses a pronoun (like "she") to continue a discussion about a newly introduced entity.
Methodology
The authors' system, developed for the TREC Novelty Track, uses a multi-layered approach to classify sentences:
- Weighted Novelty Scoring: Instead of treating all words equally, the system assigns different weights to "novel" terms based on their category (e.g., person, location, organization, or verb). These weights are learned automatically from training data.
- Focus Tracking: This is the core innovation. The system maintains a "focus variable" that tracks whether the previous sentence was novel. If a new segment starts, the system expects a burst of novelty.
- Linguistic Heuristics: The system uses a finite-state parser to identify sentences starting with third-person pronouns. If a sentence lacks "new" keywords but starts with "She" or "It" and the previous context was novel, the system assumes the focus remains on that novel information.
- Randomized Hill-Climbing Learner: To tune the 11 real-valued parameters (weights and thresholds), the authors used a randomized hill-climbing algorithm with backtracking. This allowed the system to optimize for specific goals, such as maximizing precision over the standard F-measure.
- Vector-Space Baseline: They also implemented a standard Cosine similarity model to serve as a baseline for comparison.
The researchers utilized data from the Text Retrieval Conference (TREC) Novelty Track, using the 2003 data for training and the 2004 data (Task 2) for evaluation.
Key Insights: The "Bursty" Nature of Novelty
A critical observation made by the authors is that novelty isn't distributed randomly; it appears in clusters or "bursts." Analysis of the training data revealed that over half of the novel sentences appeared in sequences of two or more.
By tracking the "focus" of the discourse, the system can correctly classify sentences that are conceptually new even if they are lexically "thin" (e.g., sentences with high anaphora). This context-aware approach helps break through the "precision ceiling" where stricter thresholds in traditional models often cause recall to plummet without significant precision gains.
Results
The system's performance at TREC 2004 was standout, particularly in precision-oriented tasks.
As shown in the results table, the "Prec2" run achieved a precision of 0.61, significantly higher than the participant average of 0.46. An analysis showed that the focus variable alone improved F-measure scores by approximately 5%. By tuning the $\beta$ parameter in the F-measure calculation during training, the learner was successfully "coaxed" to prioritize precision, which is often more valuable for summarization tasks than raw recall.
Conclusion and Future Work
While the system achieved record precision, the authors acknowledge several areas for improvement. The current evaluation was performed on Task 2, where relevance is pre-determined; real-world applications require solving Task 1 (finding relevant and novel sentences simultaneously). Future work could involve better handling of "nominal references" (e.g., linking "the president" to "the leader") and moving toward more robust probabilistic frameworks like Conditional Random Fields (CRFs) to better handle the sequential nature of discourse.
Summary updated