Diamonds in the Rough: Event Extraction from Imperfect Microblog Data

Ander Intxaurrondo, Eneko Agirre, Oier Lopez de Lacalle, Mihai Surdeanu

Abstract

Information extraction (IE) from microblogs like Twitter is notoriously difficult due to the noisy, approximate, and ambiguous nature of the data. This paper introduces a robust framework that adapts standard distant supervision (DS) techniques for these "imperfect" environments. By implementing approximate matching for training labels and aggregating features across multiple tweets, the researchers achieved a 33% relative improvement in F1 score over baseline systems. The results demonstrate that microblogs can effectively supplement curated knowledge bases, reaching nearly 88% of the performance of models trained on expensive manual annotations.

Introduction

Traditional Information Extraction (IE) has focused on polished newswire articles where facts are clearly stated and generally accurate. In contrast, microblogs like Twitter offer "near real-time" updates but present significant hurdles for standard algorithms. The authors identify two primary challenges: inaccuracy (approximation) and ambiguity.

A tweet might report an earthquake magnitude as 7.1, while the official record says 7.3. Standard distant supervision, which relies on exact matches to label training data, would ignore this tweet entirely, losing valuable training signal. Furthermore, the brevity of tweets often leads to ambiguity; a mention of "Peru" might refer to the epicenter or simply an affected region.

Examples of approximate, ambiguous, and missing information in microblog data.

Methods

The researchers enhanced Distant Supervision (DS)—a technique using a Knowledge Base (KB) to automatically label unannotated text—with three key innovations:

  • Approximate Distant Supervision: Instead of requiring exact string matches, the system uses similarity functions. For example, a tweet mentioning a "12km" depth can be used as a positive training example for a KB entry of "10km".
  • Feature Aggregation: To combat brevity, the system pools features from all tweets associated with a specific event. This joint classification allows the model to use combined linguistic evidence to resolve ambiguities.
  • Lenient Evaluation: The authors proposed a "soft" F1 metric that provides partial credit for values close to the truth, acknowledging that near-misses (e.g., a magnitude off by 0.1) still hold utility for end-users.

Experimental Setup

The study focused on the domain of earthquakes, involving up to 20 distinct arguments such as date, magnitude, and casualties. The Knowledge Base was constructed from 108 significant 21st-century earthquakes sourced from Wikipedia, while the Twitter data consisted of 7,841 relevant tweets. To establish a performance ceiling, 17,672 mentions were manually annotated for comparison.

Results and Discussion

The primary finding was that the "false negative" bottleneck—missing annotations due to strict matching—was the biggest barrier to performance. Standard systems discarded nearly 30% of useful training data because of minor approximations in user reports.

Precision-Recall curves comparing the baseline (DS-CRF), the aggregated model (DS^aggr-CRF), and the combined model (DS^comb-CRF).

As shown in the Precision-Recall curves, the combined model (approximate matching + feature aggregation) significantly outperforms the baseline, yielding a 33% relative improvement in F1 score. Notably, the automated system reached approximately 88% of the performance of the manually annotated ceiling, suggesting that DS is a viable, low-cost alternative for complex extraction in noisy domains.

Conclusion

While highly effective, the system's similarity thresholds are currently domain-specific to earthquakes. Future work will explore more general event-detection algorithms and multi-instance multi-label strategies to resolve numeric collisions, such as when magnitude and depth values overlap. Overall, this research proves that by embracing the "slop" of social media rather than fighting it, we can extract structured, high-quality event data from the chaos of Twitter.

Summary updated

Abstract

We introduce a distantly supervised event extraction approach that extracts complex event templates from microblogs. We show that this near real-time data source is more challenging than news because it contains information that is both approximate (e.g., with values that are close but different from the gold truth) and ambiguous (due to the brevity of the texts), impacting both the evaluation and extraction methods. For the former, we propose a novel, “soft”, F1 metric that incorporates similarity between extracted fillers and the gold truth, giving partial credit to different but similar values. With respect to extraction methodology, we propose two extensions to the distant supervision paradigm: to address approximate information, we allow positive training examples to be generated from information that is similar but not identical to gold values; to address ambiguity, we aggregate contexts across tweets discussing the same event. We evaluate our contributions on the complex domain of earthquakes, with events with up to 20 arguments. Our results indicate that, despite their simplicity, our contributions yield a statistically-significant improvement of 33% (relative) over a strong distantly-supervised system. The dataset containing the knowledge base, relevant tweets and manual annotations is publicly available.
BibTeX
@inproceedings{intxaurrondo2015diamonds,
  author = {Ander Intxaurrondo and Eneko Agirre and Oier Lopez de Lacalle and Mihai Surdeanu},
  title = {{Diamonds in the Rough: Event Extraction from Imperfect Microblog Data}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2015},
  doi = {10.3115/v1/n15-1066},
}
Figure