Cross-lingual Slot Filling from Comparable Corpora

Matthew G. Snover, Xiang Li, Wen-Pin Lin, Zheng Chen, Suzanne Tamang, Mingmin Ge, Adam Lee, Qi Li, Hao Li, Sam Anzaroot, Heng JiBUCC @ ACL 2011 · OpenReview

TL;DR

This paper tackles the challenging task of cross-lingual slot filling, aiming to extract information about entities from foreign language documents and present it in English. They introduce a novel supervised rescoring approach that leverages global evidence from comparable corpora, significantly improving performance over existing methods and even outperforming monolingual pipelines on larger datasets.

What Problem Does This Paper Address?

The traditional slot filling task, as seen in NIST's TAC KBP track, focuses on extracting specific attributes (like "title" or "spouse" for a person) for entities from text. However, these efforts have been largely confined to monolingual data. As the proportion of non-English content on the web grows, relying solely on English data limits the ability to build comprehensive entity profiles. This paper addresses this limitation by proposing a cross-lingual slot filling task: extracting information for an English query from comparable corpora (documents on similar topics but not direct translations) in a foreign language (specifically Chinese in this work) and presenting the results in English. This is inherently difficult due to the combined challenges of information extraction (IE) and machine translation (MT).

Methods

The authors develop several baseline pipelines to tackle this cross-lingual problem. These fall into two main categories:

  • Type A: Translate then Fill: Chinese documents are translated into English, and then standard English slot filling systems are applied.
  • Type B: Fill then Translate: English queries are translated into Chinese, Chinese slot filling systems are used, and then the extracted answers are translated back into English.

They also explore monolingual slot filling on English data as a benchmark. A key innovation is their supervised rescoring approach. This method aims to improve the precision of the extracted slot fills by:

  1. Over-generating candidates: Combining the outputs of multiple weak baseline pipelines to ensure high recall.
  2. Global Validation: Using a large collection of comparable corpora to validate these candidate answers. This involves looking for evidence that an answer frequently appears with the query in various contexts (e.g., co-occurrence, coreference, relations) across documents and languages.
  3. Supervised Rescoring: Training a Maximum Entropy model using a rich set of features (local and global, shallow and deep, across languages) to rescore the candidate answers and select the most confident ones.

Datasets

  • English TDT5: A collection of 278,358 documents used for training and evaluation.
  • Chinese TDT5: A collection of 56,424 documents used as the source for cross-lingual data.
  • Gigaword: Used for English global statistics.
  • DARPA GALE MT training corpora: Used for English global statistics.

The paper does not introduce a new benchmark dataset but rather adapts existing ones for the cross-lingual slot filling task.

Key Insights and Ideas

  • Complementarity of Monolingual and Cross-lingual Pipelines: Even with smaller datasets, cross-lingual pipelines can extract a comparable number of correct answers to monolingual pipelines on much larger datasets, highlighting the value of foreign language data.
  • MT Errors are a Major Hurdle: Machine translation errors significantly impact the performance of Type A pipelines, leading to incorrect query translations or answer translations.
  • Global Evidence is Powerful: Leveraging information redundancy across a large collection of comparable documents is crucial for validating answers and improving precision, especially for cross-lingual tasks.
  • Over-generation and Rescoring is Effective: Combining multiple weak systems to generate a broad set of candidates and then using a sophisticated rescorer to filter them is a winning strategy.
  • Feature Engineering Matters: A diverse set of features, including local context, global document statistics, and fact-based knowledge, are essential for effective cross-lingual slot filling.

Key Results

The proposed supervised rescoring approach, using global evidence from comparable corpora, achieved a 38.5% relative improvement in Precision and an 86.7% relative improvement in Recall over several state-of-the-art approaches. Notably, their final system outperformed monolingual slot filling pipelines that were trained on much larger monolingual corpora. The union of all baseline pipelines also showed improvements over individual baselines, but the rescoring method further enhanced performance.

Limitations and Future Work

  • Data Scarcity: The authors acknowledge that the limited amount of training data for their rescoring model can lead to overfitting.
  • Annotation Cost: Manual annotation for training and evaluation is costly, motivating future work in active learning and semi-supervised learning.
  • Language Independence: While demonstrated on English-Chinese, the framework is designed to be language-independent, with a future goal of extending it to more languages.
  • Improving MT: The authors plan to use their results to provide feedback for improving MT performance, particularly for fact extraction.
  • Feature Engineering: Further feature engineering is suggested to address remaining errors and improve performance.

Summary updated

BibTeX
@inproceedings{snover2011cross,
  author = {Matthew G. Snover and Xiang Li and Wen-Pin Lin and Zheng Chen and Suzanne Tamang and Mingmin Ge and Adam Lee and Qi Li and Hao Li and Sam Anzaroot and Heng Ji},
  title = {{Cross-lingual Slot Filling from Comparable Corpora}},
  booktitle = {BUCC @ ACL},
  year = {2011},
  url = {https://openreview.net/forum?id=qzyAiAHz6sn},
}
Figure