Introduction
Traditional machine translation (MT) evaluation metrics, such as BLEU or HTER, often fail to capture the practical utility of a translation within a larger automated pipeline. A translation might be grammatically imperfect yet still allow a downstream task, such as sentiment analysis, to function correctly. Conversely, a single mistranslated word in a technical or legal context can be catastrophic despite a high BLEU score. This paper proposes a shift from general quality estimation to task-specific "acceptability."
Methodology
Defining Acceptability
The authors define a translation $t$ as acceptable for a specific downstream task $f$ if the output of the task on the translation matches the output on the gold-standard reference $r$. Formally, this is expressed as $f(t) = f(r)$. This definition allows for a binary classification of translation quality based on functional consistency rather than linguistic perfection.
Automatic Annotation
A key contribution of this work is the method for generating large-scale training data without human intervention. By using existing parallel corpora (source $s$, reference $r$), the authors generate translations $t$ using an MT system. They then apply a downstream NLP tool (e.g., a sentiment classifier or NER tagger) to both $t$ and $r$. If the outputs are identical, the pair $(s, t)$ is labeled as "acceptable"; otherwise, it is "unacceptable."
Detection Models
To predict acceptability at inference time (where the reference $r$ is unavailable), the authors developed a neural architecture called BiRNN. This model employs separate Bidirectional GRU layers to encode the source and target sentences. An attention mechanism integrates these representations into a single vector, which is then processed through non-linear layers to output a probability of acceptability.
Experimental Results
The framework was evaluated across three downstream tasks: subjectivity classification, sentiment analysis, and Named Entity Recognition (NER). The experiments utilized WMT18 Chinese-English data to train MT systems of varying quality levels.
Performance Comparison
The BiRNN model consistently outperformed both the baseline (which assumes all translations are acceptable) and a feature-based model (BiQuEst). The gains were particularly pronounced in the NER task, which proved to be highly sensitive to translation errors. For the weakest MT system tested, BiRNN improved detection accuracy for NER by approximately 25% absolute over the baseline.
Data Scaling and Task Sensitivity
The study found that the performance of the acceptability detector scales well with data, showing continued improvements up to 5 million training instances. Furthermore, the results highlighted that different tasks have different tolerance levels for MT errors. While sentiment analysis is relatively robust to minor translation flaws, NER acts as a "canary in the coal mine," frequently failing when even minor entities are mistranslated.
Conclusion
By reframing MT quality as task-specific acceptability, this research provides a scalable and automated way to build quality detectors. This approach leverages the abundance of parallel data to create models that are aware of the requirements of downstream applications, offering a more meaningful metric for real-world NLP pipelines. Future work may involve integrating these detectors directly into the MT training process to optimize for downstream success.
Summary updated