TL;DR
This paper introduces SIMR-cl, a novel algorithm that automatically filters out non-parallel texts from collections intended for machine translation training. By modifying the existing SIMR bitext mapping algorithm with a chain length constraint, SIMR-cl achieves high accuracy in distinguishing comparable texts from truly parallel ones, a crucial step for improving the quality of statistical machine translation systems.
What Problem Does This Paper Address?
Modern Statistical Machine Translation (SMT) systems rely heavily on large amounts of parallel text (sentences that are direct translations of each other) for training. However, automatically collected parallel corpora often contain "comparable" texts – documents on the same topic but not direct translations. This "noisy" data can degrade SMT performance. Existing methods for aligning parallel texts often fail to warn users when the input texts are not truly parallel. This paper tackles the challenging problem of distinguishing between parallel texts and comparable texts, which are not alignable at the sentence level, to create cleaner training data for SMT.
Methods
The core of the paper's contribution lies in a modification of the SIMR (Sentence-level Information Mapping and Retrieval) bitext mapping algorithm. SIMR works by identifying points of correspondence between two texts and attempting to construct piecewise linear approximations of a "True Bitext Map." The authors introduce a new parameter to SIMR: a maximum chain length. This constraint limits the length of the correspondence chains SIMR accepts. The intuition is that in comparable texts, spurious points of correspondence will tend to form longer, less linear chains than those found in truly parallel texts. This modified algorithm is named SIMR-cl.
The training of SIMR-cl involves a state-space search algorithm (simulated annealing) to find optimal parameter values. The objective function aims to find a threshold such that parallel texts have a high "SIMR-cl score" (defined as bitext map density) and non-parallel texts have a low score. The error is measured using a concept similar to "margin slack" from support vector machines.
Datasets
- STRAND corpus: A collection of 326 French/English candidate bitexts used for evaluating the initial task of discriminating parallel texts from unrelated texts.
- Romanian/English Multext-East 1984 corpus: Used to simulate comparable texts by pairing non-aligned, consecutive chunks of varying lengths (10, 50, and 100 segments). This corpus was also used to train and test SIMR-cl.
- Hindi/English data from the "Surprise Language Exercise": A collection of documents from Indian government web pages, used to test the method's performance on a new language pair and genre without specific parameter optimization.
Key Insights and Ideas
- Comparable texts are a significant problem for SMT: The paper highlights a real-world issue where even state-of-the-art SMT systems were trained on data that was not truly parallel, impacting their performance.
- Chain length as a discriminator: The key insight is that limiting the length of correspondence chains in bitext maps can effectively differentiate between parallel and comparable texts. Longer, less linear chains are more indicative of comparable texts.
- SIMR-cl's robustness: The authors present evidence that SIMR-cl's parameters are not overly sensitive to specific language pairs or text genres, suggesting its general applicability.
- Bitext map density as a proxy for accuracy: The paper suggests that the density of a bitext map is highly correlated with its accuracy, a finding that could lead to training bitext mapping algorithms without hand-aligned data.
Key Results
- STRAND data: SIMR-cl achieved 96% accuracy in distinguishing parallel from unrelated texts, matching the performance of the tsim algorithm when trained on a sufficient amount of data.
- Romanian/English comparable texts: SIMR-cl consistently outperformed the tsim algorithm across various text lengths, achieving accuracies of 90%, 96.5%, and 98.5% for texts of 10, 50, and 100 segments, respectively, compared to tsim's consistent 66%.
- Hindi/English Surprise Language Data: SIMR-cl successfully filtered out 6 document pairs (accounting for 7% of the English word count) that were deemed comparable but not parallel by a human speaker. The paper also noted an interesting phenomenon where some "translated" sections exhibited summarization and detail omission.
Limitations and Future Work
- Recall evaluation: The authors acknowledge that a rigorous evaluation of recall (ensuring all non-parallel segments were identified) would be extremely time-consuming and was not fully performed.
- Omissions in translations: The paper suggests combining SIMR-cl with methods for detecting omissions in translations to further improve translation quality.
- Machine learning for translation with summarization: The discovery of data exhibiting summarization in translations opens avenues for machine learning approaches to translation that account for this phenomenon.
- Training without hand-aligned data: The strong correlation between bitext map density and accuracy suggests the possibility of training bitext mapping algorithms using map density as an objective function, eliminating the need for manual alignment.
Summary updated