Abstract
This paper argues that standard evaluations for topic tracking are unrealistic because they assume either zero user interaction or a user who is an "always-on" oracle. The authors propose a more realistic "Interactive Tracking" model where users check news in batches, get fatigued, and provide limited feedback, demonstrating that systems can achieve high performance with significantly less user effort.
Introduction
In the early 2000s, two major frameworks dominated information retrieval: TREC Information Filtering and Topic Detection and Tracking (TDT). TREC assumed a "perfect" user (an oracle) was always available to judge every document flagged as relevant. Conversely, TDT assumed systems must operate in total isolation after seeing only a few initial examples.
Neither reflects how humans actually consume news. Real users check news at discrete intervals, experience fatigue, and lose interest when presented with irrelevant content. This paper bridges this gap by creating an evaluation framework that incorporates these human constraints—like batching and fatigue—without sacrificing tracking accuracy.
The Interactive Tracking Model
The authors introduce a decision-theoretic model for Interactive Tracking. For every incoming document, the system chooses one of four actions based on similarity to the current topic:
- Silent Ignore: Label it non-relevant.
- Mark Relevant: Label it relevant without updating the topic model.
- Autonomous Adapt: Label it relevant and update the model without user input.
- Query User: Ask the user for a judgment and adapt based on the answer.
To increase realism, the system utilizes a session buffer. Instead of processing documents one-by-one, it collects them and presents them to the user at discrete intervals. User fatigue is modeled by simulating users who stop reading after seeing a certain number of irrelevant documents or who only agree to judge a fixed number of articles per session.
Experimental Setup
The study utilized standard benchmarks from the Linguistic Data Consortium (LDC):
- TDT2: Used for training system parameters.
- TDT3/1 and TDT3/2: Used for evaluation, consisting of manually transcribed news stories in English and Mandarin from sources like AP, CNN, and the New York Times.
Key Results and Insights
The interactive system demonstrated that tracking performance can be significantly improved with minimal user effort. Key findings include:
- Cost Reduction: The Interactive Tracking system showed a 48.5% improvement in tracking cost over the simple, non-interactive TDT baseline.
- Efficiency: It achieved results nearly identical to the "perfect" filtering model but with a 25.9% reduction in the number of feedback requests sent to the user.
- Fatigue Resilience: In simulations where users limited themselves to only 3 judgments per session, the user effort (activity) dropped by 66%, while the tracking cost only increased by a manageable 16.9%.
- Batching and Ordering: Surprisingly, waiting to show a user a batch of documents (even weekly) had almost no negative impact on tracking cost. While sorting the buffer by relevance score improves user precision, sorting by "action" (prioritizing documents the system is unsure about) helps the system learn faster.
Conclusion
This research demonstrates that modeling realistic human behavior—such as batching and fatigue—does not necessarily compromise topic tracking performance. By allowing for autonomous adaptation and strategic user queries, systems can remain effective with significantly reduced human effort. Future work may explore incorporating negative feedback for adaptation and conducting studies with actual human users to further refine these simulations.
Summary updated