Improving realism of topic tracking evaluation

Anton Leuski, James Allan

Abstract

This paper argues that standard evaluations for topic tracking are unrealistic because they assume either zero user interaction or a user who is an "always-on" oracle. The authors propose a more realistic "Interactive Tracking" model where users check news in batches, get fatigued, and provide limited feedback, demonstrating that systems can achieve high performance with significantly less user effort.

Introduction

In the early 2000s, two major frameworks dominated information retrieval: TREC Information Filtering and Topic Detection and Tracking (TDT). TREC assumed a "perfect" user (an oracle) was always available to judge every document flagged as relevant. Conversely, TDT assumed systems must operate in total isolation after seeing only a few initial examples.

Neither reflects how humans actually consume news. Real users check news at discrete intervals, experience fatigue, and lose interest when presented with irrelevant content. This paper bridges this gap by creating an evaluation framework that incorporates these human constraints—like batching and fatigue—without sacrificing tracking accuracy.

The Interactive Tracking Model

The authors introduce a decision-theoretic model for Interactive Tracking. For every incoming document, the system chooses one of four actions based on similarity to the current topic:

  1. Silent Ignore: Label it non-relevant.
  2. Mark Relevant: Label it relevant without updating the topic model.
  3. Autonomous Adapt: Label it relevant and update the model without user input.
  4. Query User: Ask the user for a judgment and adapt based on the answer.
Flowchart of the Interactive Tracking model incorporating session buffers and user interest modeling.

To increase realism, the system utilizes a session buffer. Instead of processing documents one-by-one, it collects them and presents them to the user at discrete intervals. User fatigue is modeled by simulating users who stop reading after seeing a certain number of irrelevant documents or who only agree to judge a fixed number of articles per session.

Experimental Setup

The study utilized standard benchmarks from the Linguistic Data Consortium (LDC):

  • TDT2: Used for training system parameters.
  • TDT3/1 and TDT3/2: Used for evaluation, consisting of manually transcribed news stories in English and Mandarin from sources like AP, CNN, and the New York Times.

Key Results and Insights

The interactive system demonstrated that tracking performance can be significantly improved with minimal user effort. Key findings include:

  • Cost Reduction: The Interactive Tracking system showed a 48.5% improvement in tracking cost over the simple, non-interactive TDT baseline.
  • Efficiency: It achieved results nearly identical to the "perfect" filtering model but with a 25.9% reduction in the number of feedback requests sent to the user.
Performance comparison of tracking systems across TDT3 datasets.
  • Fatigue Resilience: In simulations where users limited themselves to only 3 judgments per session, the user effort (activity) dropped by 66%, while the tracking cost only increased by a manageable 16.9%.
  • Batching and Ordering: Surprisingly, waiting to show a user a batch of documents (even weekly) had almost no negative impact on tracking cost. While sorting the buffer by relevance score improves user precision, sorting by "action" (prioritizing documents the system is unsure about) helps the system learn faster.

Conclusion

This research demonstrates that modeling realistic human behavior—such as batching and fatigue—does not necessarily compromise topic tracking performance. By allowing for autonomous adaptation and strategic user queries, systems can remain effective with significantly reduced human effort. Future work may explore incorporating negative feedback for adaptation and conducting studies with actual human users to further refine these simulations.

Summary updated

Abstract

Topic tracking and information filtering are models of interactive tasks, but their evaluations are generally done in a way that does not reflect likely usage. The models either force frequent judgments or disallow any at all, assume the user is always available to make a judgment, and do not allow for user fatigue. In this study we extend the evaluation framework for topic tracking to incorporate those more realistic issues. We demonstrate that tracking can be done in a realistic interactive setting with minimal impact on tracking cost and with substantial reduction in required interaction.
BibTeX
@inproceedings{leuski2002improving,
  author = {Anton Leuski and James Allan},
  title = {{Improving realism of topic tracking evaluation}},
  booktitle = {International ACM SIGIR Conference on Research and Development in Information Retrieval},
  year = {2002},
  doi = {10.1145/564376.564394},
}
Figure