TL;DR
This paper introduces a novel "bootstrapping" approach to automatically discover parallel sentences for machine translation (MT) training, using only comparable corpora (documents on similar topics, not direct translations). Their method outperforms existing MT systems trained on clean parallel data, demonstrating the power of leveraging readily available comparable data.
What Problem Does This Paper Address?
Building high-quality statistical machine translation (SMT) systems heavily relies on large amounts of parallel text – documents that are direct translations of each other. However, creating such parallel corpora is expensive and time-consuming, especially for new languages or specialized domains. Researchers have explored using "comparable corpora" (multilingual documents that are topically aligned but not necessarily translations) as a cheaper alternative. Previous methods for extracting parallel data from comparable corpora often required pre-existing translation dictionaries, bitext, or baseline MT systems for initialization. This paper tackles the challenge of building MT systems from comparable corpora without relying on any of these initial bilingual resources, using only publication dates and overlapping vocabulary as weak signals.
Methods
The core of the paper is a two-stage bootstrapping approach:
- Document Translation Pair Discovery:
- Overlapping Cosine Distance (OCD): This initial step uses a set of words that appear in both languages (overlapping vocabulary) as features. Documents are represented as tf.idf vectors based on these overlapping words. Cosine distance is used to find similar document pairs.
- Length and Diversity Filtering: To refine the OCD results, they apply length filtering (target document length within 20% of source) and diversity filtering (limiting the number of target documents considered for each source document). This helps to remove spurious matches and introduce topic diversity.
- Sentence Extraction and MT System Training:
- Polylingual Topic Model (PLTM): A multilingual topic model (based on LDA) is trained on sentence-aligned portions of the candidate document pairs identified in the first stage. This model represents documents in a shared topic space.
- Online Variational Bayes (VB): To handle large corpora like Gigaword efficiently, they adapt the PLTM inference to an online VB approach, which is faster than traditional Gibbs sampling.
- Approximate Nearest Neighbor (NN) Search: For efficient similarity comparisons in the topic space, they use approximate NN search techniques (k-d trees) to compute Hellinger (He) distances, which are related to Euclidean distances.
- Sentence Alignment and MT Training: The top-ranked document pairs are then used to extract and align sentence pairs, which form the training data for a Moses SMT system.
Datasets
- Training Data:
- English and Spanish Gigaword news stories (specifically, four years of data: 2010, 2009, 2005, and 2004).
- WMT'11 parallel corpora: News Commentary (NC) and Europarl (EP) were used for baseline MT systems.
- Evaluation Data:
- WMT'11 News test set.
- WMT'08 Europarl test set.
Key Insights and Ideas
- Bootstrapping without Prior Knowledge: The most significant insight is the ability to build a functional MT system by bootstrapping from scratch using only comparable corpora, without needing any initial bilingual dictionaries or parallel seed data. This dramatically lowers the barrier to entry for low-resource languages.
- Leveraging Weak Signals: The approach effectively utilizes weak signals like publication dates (implicitly through same-day news) and overlapping vocabulary to identify translation candidates.
- Topic Models for Cross-Lingual Alignment: Using polylingual topic models to represent documents in a shared semantic space is a powerful way to bridge the language gap and enable cross-lingual similarity comparisons.
- Efficiency Matters: The adoption of online VB for PLTM and approximate NN search highlights the practical necessity of efficient algorithms for processing massive datasets.
Key Results
- Outperforming Baselines: MT systems trained on parallel sentences extracted from the Gigaword comparable corpus (using their bootstrapping method) outperformed baseline MT systems trained on clean, official WMT'11 parallel data (News Commentary).
- For the WMT'11 News test set, their GW-trained system achieved a BLEU score of 24.92, compared to 23.75 for the NC baseline.
- When combined with WMT'11 NC data, their system achieved 25.90.
- Comparable Performance on In-Domain Data: On the WMT'08 Europarl test set, their system trained on extracted EP+GW data achieved a BLEU score of 31.59, which is competitive with the baseline trained on clean EP data (32.06).
- Ablation Studies: Deduplication of extracted sentence pairs had mixed effects, sometimes hurting performance and sometimes boosting it, suggesting that redundancy might be beneficial for certain topic models or that careful deduplication strategies are needed.
Limitations and Future Work
- Domain Specificity: The performance of the extracted parallel data can be sensitive to the domain of the comparable corpus and the test set. The paper notes that EP is out-of-domain for the News test set.
- Redundancy Effects: The interaction between data redundancy, deduplication, and the PLTM model's performance warrants further investigation.
- Future Work: The authors suggest further studies into the interactions of test domain, redundancy, and model settings. The core approach itself opens avenues for applying this bootstrapping method to a wider range of languages and domains where parallel data is scarce.
Summary updated