Translationese-index: Using Likelihood Ratios for Graded and Generalizable Measurement of Translationese

Yikang Liu, William Zhang, Yiming Wang, Jialong Tang, Pei Zhang, Baosong Yang, Fei Huang, Rui Wang, Haiyan Hu2025 · DOI

Abstract: Yikang Liu, Wanyang Zhang, Yiming Wang, Jialong Tang, Pei Zhang, Baosong Yang, Fei Huang, Rui Wang, Hai Hu. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing.

Introduction

Translated text often exhibits "translationese"—subtle linguistic markers that distinguish it from native-authored text. While grammatically correct, these outputs can sound unnatural or "off," creating an uncanny valley for Large Language Models (LLMs) that ingest machine-translated data. Historically, translationese detection has been treated as a binary classification problem. However, this approach fails to account for the graded nature of stylistic naturalness and often suffers from poor generalizability, as classifiers frequently latch onto topic or genre rather than pure style.

This paper introduces the T-index (Translationese-index), a novel metric designed to provide a continuous, graded measurement of translationese that remains robust across different domains and models.

Methodology: The T-index Framework

The T-index leverages Likelihood Ratios (LLR) to isolate stylistic features from content. The process involves three primary stages: synthesizing paired data, training contrastive scoring models, and computing the final index.

The T-index pipeline: (I) Synthesizing paired translations, (II) Training contrastive SFT scoring models, and (III) Computing the translationese index via likelihood ratios.
  1. Synthesis: Researchers generate paired translations for the same source text $x$, forcing a model to produce both a "high-translationese" (literal) version $\tilde{y}$ and a "low-translationese" (idiomatic) version $y$.
  2. Scoring Models: Two small language models (0.5B parameters) are fine-tuned. Model $\tilde{\theta}$ is trained on literal samples, while model $\theta$ is trained on idiomatic ones.
  3. Index Computation: The T-index for any given translation is calculated as the difference in log-probabilities between the two models: $$\text{T-index} \approx \log P_{\tilde{\theta}}(y|x) - \log P_{\theta}(y|x)$$

By subtracting these log-probabilities, the components related to the text's topic or specific vocabulary (the "what") are canceled out, leaving a pure signal of the translation style (the "how").

Experimental Setup and Datasets

The metric was validated using a mix of synthetic and real-world data:

  • Synthetic Benchmark: 14,000 sentences from 19th-century novels and the COCA corpus, translated into Chinese using Qwen2.5 and Llama3.3 with specific stylistic prompts.
  • Human Annotations: 150 translations from seven different MT/LLM systems (including Google Translate and DeepSeek-R1) were rated for naturalness by 30 translation studies experts.
  • Standard Benchmarks: Tests on Flores101, LCMC, and ZCTC datasets evaluated the metric's correlation with existing Quality Estimation (QE) tools.

Key Results

Generalization and Robustness

The T-index demonstrated exceptional generalizability. In binary classification tasks, it achieved 95.8% accuracy in-domain and maintained over 90% accuracy when tested on entirely different genres. Unlike standard supervised classifiers, the T-index does not degrade when moving from fiction to news, confirming its ability to ignore domain-specific confounding factors.

Alignment with Human Judgment

The metric shows a strong alignment with expert human intuition. It achieved a Pearson correlation of $r = 0.418$ with human pointwise ratings. In pairwise comparisons—where the model must choose which of two sentences sounds more like a translation—the T-index agreed with unanimous human judgments 84.6% of the time.

Pointwise-Pairwise agreement between the T-index and human ratings, showing higher agreement as human consensus increases.

A Distinct Dimension of Quality

Analysis revealed that the T-index has a very low correlation with traditional metrics like BLEU or COMET. This indicates that "naturalness" is a unique quality dimension currently overlooked by standard evaluation tools, making the T-index a valuable complementary metric for Machine Translation Quality Estimation.

Conclusion

The T-index provides a scalable, data-efficient, and highly generalizable method for measuring translationese. By moving beyond binary classification to a continuous likelihood-based scale, it offers researchers a precise tool to monitor and improve the stylistic naturalness of LLM-generated content and machine translation systems.

Summary updated

BibTeX
@misc{liu2025translationese,
  author = {Yikang Liu and William Zhang and Yiming Wang and Jialong Tang and Pei Zhang and Baosong Yang and Fei Huang and Rui Wang and Haiyan Hu},
  title = {{Translationese-index: Using Likelihood Ratios for Graded and Generalizable Measurement of Translationese}},
  year = {2025},
  doi = {10.18653/v1/2025.emnlp-main.633},
}
Figure