An Unsupervised Method for Automatic Translation Memory Cleaning

Masoud Jalili Sabet, Matteo Negri, Marco Turchi, Eduard BarbuACL 2016 · DOI

Introduction

In professional translation, Translation Memories (TMs) are essential databases containing pairs of source and target sentences, known as Translation Units (TUs). While TMs significantly enhance translator productivity, they are often "polluted" with noise, especially in large-scale, collaboratively-built repositories like MyMemory. This noise includes misspellings, formatting errors, and incorrect translations.

Examples of noisy translation units in the MyMemory dataset, showing issues like casing, missing punctuation, and semantic mismatches.

Manual cleaning of these millions of entries is economically unfeasible. This paper presents a fully unsupervised approach to automatically identify and remove low-quality TUs, bypassing the need for expensive human-labeled training data.

Methodology

The proposed system utilizes an unsupervised workflow based on "feature triangulation."

1. Feature Engineering

The researchers defined 31 features across three independent categories:

  • Basic Features: Surface-level checks (e.g., number/date matching, language ID).
  • QE-derived Features: Quality Estimation metrics focusing on word alignment and semantic adequacy.
  • Word Embeddings: Semantic checks using cross-lingual word embeddings to assess conceptual alignment between source and target.

2. Label Inference and Ensemble Training

To generate training data without human labels, the method employs a triangulation strategy. Two feature groups (e.g., Basic and QE) are used to rank a subset of the TM. The top $k$ pairs are labeled as "good" and the bottom $k$ as "bad."

Crucially, an ensemble of Extremely Randomized Trees is then trained using only the third feature group (e.g., Word Embeddings). This "feature swapping" ensures the model learns the underlying concept of quality rather than simply replicating the ranking heuristic of the other groups.

Experimental Setup

The method was evaluated on the English-Italian portion of the MyMemory dataset.

  • Training Data: Unlabeled pools ranging from 50,000 to 1 million TUs.
  • Evaluation: A "gold standard" test set of 1,000 TUs manually annotated by native speakers.
  • Metric: Balanced Accuracy (BA) was used to account for the typical imbalance between high-quality and low-quality units in TMs.

Results and Discussion

The unsupervised ensemble demonstrated remarkable performance, consistently outperforming traditional machine-translation-based baselines.

Balanced Accuracy as a function of training set size (k) and pool size (Z). The unsupervised approach (Z=50K) exceeds the performance of the supervised Barbu15 method.

The Polarization Paradox

A key finding was that smaller data pools ($Z=50K$) for label inference often yielded better results than massive pools ($Z=1M$). In very large pools, the selected "good" and "bad" examples become so extreme (highly polarized) that the classifier fails to generalize to the "gray area" cases encountered in real-world data.

Comparison with Supervised Methods

In its optimal configuration ($Z=50K, k=15K$), the unsupervised method achieved a Balanced Accuracy of approximately 0.79. This not only beat the MT-based baseline but also outperformed the state-of-the-art supervised method (Barbu, 2015), which requires human-annotated data.

Conclusion

This research demonstrates that pseudo-labels generated through feature triangulation are robust enough to train high-performing classifiers for TM cleaning. By leveraging the independence of different feature views, the system effectively automates a previously manual and costly process, rivaling the accuracy of supervised learning without the associated data collection overhead.

Summary updated

BibTeX
@inproceedings{sabet2016unsupervised,
  author = {Masoud Jalili Sabet and Matteo Negri and Marco Turchi and Eduard Barbu},
  title = {{An Unsupervised Method for Automatic Translation Memory Cleaning}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2016},
  doi = {10.18653/v1/p16-2047},
}
Figure