A Factory of Comparable Corpora from Wikipedia

Alberto Barrón-Cedeño, Cristina España-Bonet, Josu Boldoba, Lluís MàrquezBUCC @ ACL 2015 · DOI · OpenReview

TL;DR

This paper presents a method to automatically extract comparable corpora from Wikipedia, specifically tailored for domain-specific tasks. They demonstrate that these Wikipedia-derived corpora significantly improve machine translation performance for in-domain texts compared to generic corpora.

What Problem Does This Paper Address?

Creating high-quality, domain-specific comparable corpora is a significant challenge. While the web offers a vast amount of multilingual data, it's often noisy and unstructured. Existing approaches to extract comparable data from sources like Wikipedia can be complex and don't always yield corpora suitable for specific domains. This lack of specialized, high-quality parallel data hinders the performance of natural language processing (NLP) tasks, particularly machine translation (MT), which heavily relies on such resources for training.

Methods

The authors propose a pipeline for extracting domain-specific comparable corpora from Wikipedia. The core idea leverages Wikipedia's category structure.

  1. Domain Vocabulary Definition: They first define a domain vocabulary by extracting terms from articles within a chosen top-level category (e.g., "Sport"). This involves standard text pre-processing and selecting the most frequent terms.
  2. Category Graph Exploration: Using this vocabulary, they explore Wikipedia's category graph (a network of interconnected categories) starting from the root category of the domain. They employ a breadth-first search, applying heuristics to identify categories likely belonging to the target domain. This involves checking if category titles contain terms from the domain vocabulary and using depth-based filtering to manage the graph's complexity and potential noise.
  3. Parallel Sentence Extraction: Once comparable documents are identified, they extract parallel sentences by measuring the similarity between sentence pairs across languages using various metrics (character n-grams, pseudo-cognates, word n-grams, and length factor). They empirically determine thresholds for these measures to maximize F1 scores on a manually annotated dataset.

Datasets

  • Wikipedia Editions: The authors primarily use the English and Spanish Wikipedia editions.
  • Europarl v7: Used as a general-purpose parallel corpus for comparison in machine translation experiments.
  • News Commentaries 2011 & 2009: Used as out-of-domain test and development sets for MT evaluation.
  • GNOME Localization Files: Used as an additional in-domain test set for the Computer Science domain.
  • Manually Annotated Corpus: A small corpus of 30 article pairs (10 per domain) annotated at the sentence level to determine optimal similarity thresholds for parallel sentence extraction.

Key Insights and Ideas

  • Wikipedia's Category Structure is Key: The paper highlights that Wikipedia's inherent, albeit sometimes chaotic, category taxonomy can be effectively exploited to identify domain-specific content.
  • Heuristics for Navigating Wikipedia's Graph: The authors demonstrate that simple heuristics, like checking category titles against a domain vocabulary and controlling graph exploration depth, are sufficient to extract relevant comparable data.
  • Domain-Specific Corpora Outperform Generic Ones for In-Domain Tasks: The most striking insight is that even noisy, automatically extracted comparable corpora from Wikipedia significantly boost MT performance for specific domains, often surpassing large, general-domain corpora like Europarl.
  • Parallel Sentence Extraction Can Be Noisy but Effective: The parallel sentence extraction method, while not aiming for perfect precision, generates enough useful data to substantially improve MT systems. The SMT system's word alignment process can handle some of the noise.
  • Size Matters, but Domain Relevance is Crucial: While larger corpora generally lead to better results, the paper shows that smaller, domain-specific corpora derived from Wikipedia can outperform larger, general-domain corpora for in-domain translation tasks.

Key Results

  • Machine Translation Improvement: Systems trained on Wikipedia-derived parallel corpora achieved significantly higher BLEU scores on in-domain test sets compared to systems trained solely on the general Europarl corpus. For example, a specialized system for the "Science" domain achieved a BLEU score of 57.40, while the Europarl system scored 34.00.
  • Outperforming General Corpora: Even when trained with less data, Wikipedia-based systems often outperformed the general Europarl translator on in-domain tasks.
  • Robustness of the Method: The approach successfully extracted comparable corpora for Computer Science, Science, and Sports domains, demonstrating its applicability across different subject areas.
  • Parallel Sentence Extraction Yields Large Corpora: For the "Science" domain, they extracted over 1 million parallel sentence pairs, and for "Computer Science," around 200,000 pairs, which are substantial for SMT training.

Limitations and Future Work

  • Wikipedia's Category Chaos: The authors acknowledge that Wikipedia's category graph is not perfectly structured, leading to overlapping categories and potential noise. They employ heuristics to mitigate this but note that fixing the underlying issue is beyond the scope of their work.
  • Threshold Optimization: The optimal thresholds for similarity measures were determined empirically on a small annotated corpus. The choice of threshold involves a trade-off between precision and recall, depending on the intended application.
  • Future Work: The authors mention their ongoing efforts to generate more robust models for parallel sentence identification and to design other indirect evaluation schemes to validate their model's performance. They also suggest that the prototype is currently being used for other language pairs and domains.

Summary updated

BibTeX
@inproceedings{barroncedeno2015factory,
  author = {Alberto Barrón-Cedeño and Cristina España-Bonet and Josu Boldoba and Lluís Màrquez},
  title = {{A Factory of Comparable Corpora from Wikipedia}},
  booktitle = {BUCC @ ACL},
  year = {2015},
  doi = {10.18653/v1/w15-3402},
}
Figure