Acquiring entailment pairs across languages and domains: A Data Analysis

Manaal Faruqui, Sebastian PadóIWCS 2011 · OpenReview

Abstract

This study investigates the "headline-as-hypothesis" heuristic for automatically constructing Textual Entailment (TE) datasets. By analyzing news corpora in English, German, and Hindi, the researchers demonstrate that while linguistic features are remarkably consistent across languages, the heuristic's performance is highly sensitive to the journalistic domain. While factual news agencies yield high-precision entailment pairs, stylistic or opinion-based outlets present a significant challenge for automated data acquisition.

Introduction

In Natural Language Processing, Textual Entailment (TE) is a core task where a system determines if a Premise ($P$) justifies a Hypothesis ($H$). Training robust TE models requires vast amounts of annotated data, which is traditionally expensive to produce. A popular shortcut involves "harvesting" data from news articles by treating the first sentence as the Premise and the headline as the Hypothesis. However, the reliability of this heuristic across different languages and journalistic genres has remained largely unexplored. This paper evaluates the generalizability of this strategy across English, German, and Hindi, as well as across various news domains.

Methods

The researchers developed a logistic regression model to predict the validity of entailment pairs. They employed several linguistically motivated predictors to identify valid pairs:

  • Weighted Overlap: A tf-idf based measure of lexical overlap between the title and the lead sentence.
  • Strict Noun Match: A binary indicator of whether all nouns in the title are present in the sentence.
  • Punctuation: A proxy for identifying ill-formed or non-propositional titles (e.g., those containing colons or dashes).
  • Article Length: A test of whether longer articles have less representative lead sentences.
Predictor significance across English, German, and Hindi.

Datasets and Evaluation

The analysis utilized several corpora to test cross-lingual and cross-domain robustness:

  1. Reuters RCV2 (English & German): Representing factual, agency-style news.
  2. EMILLE & Webdunia (Hindi): Covering political and regional South Asian news.
  3. German Domain Comparison: A comparison between Reuters (factual), Stuttgarter Zeitung (regional), and Die Zeit (a weekly paper known for stylistic, long-form writing).

A gold-standard evaluation set was created by manually annotating 1,000 pairs for each language and domain.

Key Insights

Language Robustness

The model's behavior was remarkably consistent across the three languages. As shown in the predictor analysis, weighted overlap and punctuation were the primary drivers of performance regardless of the language. This suggests that the structural relationship between a headline and its lead sentence in factual news is a cross-linguistic phenomenon.

The Domain Barrier

While language was not a significant hurdle, the journalistic domain proved to be a major factor. The heuristic relies on headlines being factual summaries. This holds for "hard news" agencies like Reuters but fails for outlets like Die Zeit, where headlines are often designed to be evocative or "clickbait" rather than strictly summarizing the first sentence.

Precision of models across different journalistic domains.

Results

  • High Precision in Agency News: On Reuters data, the model achieved over 90% precision (at 30% recall), confirming the heuristic's utility for automated dataset construction in factual domains.
  • Domain Degradation: Performance plummeted to approximately 46% when applied to Die Zeit, highlighting the limitations of the heuristic in stylistic genres.
  • Genre Over Language: A model trained on English news performed better on German news than a model trained on German hard news performed on German opinion pieces. This indicates that the "genre" of the text is a more critical variable than the language itself.

Conclusion

The "headline-as-hypothesis" heuristic is a powerful tool for building TE datasets, provided the source material is factual news. However, its effectiveness is bound by the discourse style of the publication. Future work should explore more complex sources, such as Wikipedia or paraphrase-extraction techniques, to capture a broader range of entailment relationships that are less dependent on specific journalistic conventions.

Summary updated

BibTeX
@inproceedings{faruqui2011acquiring,
  author = {Manaal Faruqui and Sebastian Padó},
  title = {{Acquiring entailment pairs across languages and domains: A Data Analysis}},
  booktitle = {IWCS},
  year = {2011},
  url = {https://openreview.net/forum?id=rKZglfnMfEe9},
}
Figure