Abstract
This research presents a semi-supervised co-training approach for identifying sentences that describe specific events in news text. By leveraging a massive corpus of unlabeled online news summaries, the method achieves significant performance gains over supervised baselines, demonstrating that large-scale unlabeled data can effectively mitigate the data bottleneck in event identification tasks.
Introduction
In applications such as question answering and news summarization, identifying sentences that describe a specific event (e.g., "the 2008 peace talks") is a critical yet challenging task. This difficulty stems from two primary factors: the semantic gap between queries and relevant sentences, and the severe lack of labeled training data.
The Data Bottleneck
Training robust models typically requires thousands of human-labeled examples. However, manual annotation for event relevance is notoriously difficult due to low inter-annotator agreement. In a validation study using Amazon Mechanical Turk (AMT), researchers found that humans often disagree on what constitutes a relevant event description, making the collection of high-quality labels slow and expensive.
Methods
The authors employ co-training, a semi-supervised technique that uses two independent "views" of the data to iteratively teach each other:
- View 1 (Keywords): Analyzes unigram and bigram overlaps between the query and the sentence.
- View 2 (Named Entities): Focuses on matches of specific people, locations, and organizations.
Bayesian Network classifiers were selected for both views due to their ability to handle dependencies between features better than Support Vector Machines (SVMs). The model begins with a small human-labeled "seed" and iteratively labels the most confident examples from a massive pool of unlabeled news summaries.
Efficiency and Scaling
A key finding was the efficiency of "leap size"—the amount of data added in each co-training iteration. Increasing the leap size proved more effective for improving accuracy and recall than simply increasing the number of iterations, while also maintaining lower computational overhead.
Datasets
- GALE (Global Autonomous Language Exploitation): Provided the initial seed data and evaluation queries.
- Newsblaster Corpus: A novel unlabeled dataset containing approximately 2.5 million sentences from 166,435 news summaries collected between 2003 and 2011.
- AMT Labels: Used to establish the baseline difficulty and human agreement levels.
Key Results
The co-trained Bayesian Network significantly outperformed supervised baselines and alternative co-training setups. The system reached 83.98% accuracy, an 8% absolute gain over the supervised baseline (77.61%).
Recall and Data Scaling
The most dramatic improvement was seen in recall, which increased by an absolute 22%. By iteratively exploring unlabeled data, the model learned to identify relevant sentences that lacked direct keyword matches with the query. Furthermore, performance scaled positively with the volume of unlabeled data; using nine years of news summaries provided more robust results than using a single year.
Additionally, the Bayesian Network approach was found to be 55 times faster than SVM-based co-training, making it highly suitable for large-scale web data processing.
Conclusion
This work demonstrates that news summaries serve as excellent "silver" data for training event identification models. While the current model relies on simple lexical features, future work may incorporate deeper semantic information to better handle sub-events and synonyms. The success of this co-training approach suggests that massive unlabeled datasets can effectively substitute for scarce manual labels in complex NLP tasks.
Summary updated