Abstract: Yikang Liu, Wanyang Zhang, Yiming Wang, Jialong Tang, Pei Zhang, Baosong Yang, Fei Huang, Rui Wang, Hai Hu. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing.
Introduction
Translated text often exhibits "translationese"—subtle linguistic markers that distinguish it from native-authored text. While grammatically correct, these outputs can sound unnatural or "off," creating an uncanny valley for Large Language Models (LLMs) that ingest machine-translated data. Historically, translationese detection has been treated as a binary classification problem. However, this approach fails to account for the graded nature of stylistic naturalness and often suffers from poor generalizability, as classifiers frequently latch onto topic or genre rather than pure style.
This paper introduces the T-index (Translationese-index), a novel metric designed to provide a continuous, graded measurement of translationese that remains robust across different domains and models.
Methodology: The T-index Framework
The T-index leverages Likelihood Ratios (LLR) to isolate stylistic features from content. The process involves three primary stages: synthesizing paired data, training contrastive scoring models, and computing the final index.
- Synthesis: Researchers generate paired translations for the same source text $x$, forcing a model to produce both a "high-translationese" (literal) version $\tilde{y}$ and a "low-translationese" (idiomatic) version $y$.
- Scoring Models: Two small language models (0.5B parameters) are fine-tuned. Model $\tilde{\theta}$ is trained on literal samples, while model $\theta$ is trained on idiomatic ones.
- Index Computation: The T-index for any given translation is calculated as the difference in log-probabilities between the two models: $$\text{T-index} \approx \log P_{\tilde{\theta}}(y|x) - \log P_{\theta}(y|x)$$
By subtracting these log-probabilities, the components related to the text's topic or specific vocabulary (the "what") are canceled out, leaving a pure signal of the translation style (the "how").
Experimental Setup and Datasets
The metric was validated using a mix of synthetic and real-world data:
- Synthetic Benchmark: 14,000 sentences from 19th-century novels and the COCA corpus, translated into Chinese using Qwen2.5 and Llama3.3 with specific stylistic prompts.
- Human Annotations: 150 translations from seven different MT/LLM systems (including Google Translate and DeepSeek-R1) were rated for naturalness by 30 translation studies experts.
- Standard Benchmarks: Tests on Flores101, LCMC, and ZCTC datasets evaluated the metric's correlation with existing Quality Estimation (QE) tools.
Key Results
Generalization and Robustness
The T-index demonstrated exceptional generalizability. In binary classification tasks, it achieved 95.8% accuracy in-domain and maintained over 90% accuracy when tested on entirely different genres. Unlike standard supervised classifiers, the T-index does not degrade when moving from fiction to news, confirming its ability to ignore domain-specific confounding factors.
Alignment with Human Judgment
The metric shows a strong alignment with expert human intuition. It achieved a Pearson correlation of $r = 0.418$ with human pointwise ratings. In pairwise comparisons—where the model must choose which of two sentences sounds more like a translation—the T-index agreed with unanimous human judgments 84.6% of the time.
A Distinct Dimension of Quality
Analysis revealed that the T-index has a very low correlation with traditional metrics like BLEU or COMET. This indicates that "naturalness" is a unique quality dimension currently overlooked by standard evaluation tools, making the T-index a valuable complementary metric for Machine Translation Quality Estimation.
Conclusion
The T-index provides a scalable, data-efficient, and highly generalizable method for measuring translationese. By moving beyond binary classification to a continuous likelihood-based scale, it offers researchers a precise tool to monitor and improve the stylistic naturalness of LLM-generated content and machine translation systems.
Summary updated