Towards leveraging closed captions for news retrieval

Roi Blanco, Gianmarco De Francisci Morales, Fabrizio SilvestriWWW 2013 · DOI

TL;DR

This paper tackles the challenge of real-time news retrieval by matching online news articles to segments of closed captions (CC) from live TV newscasts. They propose a system that segments CC streams and formulates queries to find relevant articles quickly, balancing accuracy with timeliness.

What Problem Does This Paper Address?

Imagine you're watching a TV newscast, and a story catches your eye. You'd love to read more about it online, right now. The problem is, how do you automatically find the exact online news article that matches the specific piece of news being discussed on TV, in real-time? This is the core challenge addressed by the authors. Existing news retrieval systems often struggle with the dynamic nature of live broadcasts, the noisy and informal language of CC, and the critical need for speed. Providing an article after the news segment has passed is largely useless. The authors aim to bridge this gap by developing a system that can accurately and promptly link live TV news content to relevant online articles.

Methods

The proposed solution breaks down the problem into two main sub-tasks:

  1. CC Segmentation: The system needs to identify segments within the continuous stream of closed captions that correspond to a single, cohesive news topic. This is crucial because a single CC line might not contain enough information, and a longer segment provides better context.
  2. News Retrieval: Once a segment is identified, the system must formulate an effective query to search an underlying news retrieval engine. The goal is to find articles that precisely match the topic of the CC segment, not just loosely related ones.

The authors explore two segmentation approaches: Sliding Window (SW) and Tumbling Window (TW). They also use TF-IDF (Term Frequency-Inverse Document Frequency) to select relevant terms for query formulation. A key aspect of their work is the development of a novel evaluation metric that explicitly accounts for both the relevance of a retrieved article and the timeliness of its delivery, capturing the critical trade-off between accuracy and speed.

Datasets

The authors built a dataset consisting of:

  • Closed Captions (CC): Data from 24 hours of newscasts.
  • News Articles: A repository of news articles from Yahoo! News covering the two weeks preceding the newscast dates.

They also mention that expert evaluators were used to judge pairs of matching segments and articles, with an article being considered a positive match only if it discussed the exact same news as the CC segment.

Key Insights and Ideas

  • Timeliness is Paramount: The most significant insight is that for news retrieval from live broadcasts, speed is as critical as accuracy. An accurate article delivered too late is of little value.
  • Sliding Window Outperforms Tumbling Window: The study found that a sliding window approach for segmenting CC streams generally performed better than the tumbling window approach previously used in similar research.
  • The "Sweet Spot" for Window Size: For segmenting CC, a window size of around 30 seconds appears to be optimal. Shorter windows lack sufficient context, while longer windows can lag behind topic changes.
  • Evaluation Metrics Matter: The choice of time discount function in their evaluation metric significantly impacts how system performance is perceived. Step functions are more forgiving of late matches, while exponential functions heavily penalize lateness, forcing a stronger accuracy-timeliness trade-off.

Key Results

The preliminary experiments, evaluated using Mean Average Precision (MAP) and Precision at K (P@K) normalized against an "ORACLE" (which has perfect knowledge of segment boundaries), showed:

  • The Sliding Window (SW) approach consistently outperformed the Tumbling Window (TW) approach across various window sizes and time discount functions.
  • A 30-second window size generally yielded the best results for both SW and TW methods.
  • Different time discount functions led to varying performance profiles, highlighting the impact of how lateness is penalized. For instance, the step function showed increasing performance with larger windows, while the exponential function showed the opposite.

Limitations and Future Work

  • Limitations: The paper presents a "preliminary study" and does not describe the system currently in production at IntoNow. The proposed solution is based on keyword representation for queries, which might not capture higher-order semantics.
  • Future Work: The authors suggest that queries could be more generally comprised of different units, potentially capturing named entities or other higher-order semantic information. Further exploration of query formulation techniques beyond simple keywords is a natural next step.

Summary updated

BibTeX
@inproceedings{blanco2013towards,
  author = {Roi Blanco and Gianmarco De Francisci Morales and Fabrizio Silvestri},
  title = {{Towards leveraging closed captions for news retrieval}},
  booktitle = {The Web Conference},
  year = {2013},
  doi = {10.1145/2487788.2487853},
}
Figure