The Benefits of Being Uncertain: Perplexity as a Signal for Naturalness in Multilingual Machine Translation

Timothy Pistotti, Michael Witbrock, Dianne P. O’Leary, Jason Brown2025 · DOI

Abstract

Model-internal uncertainty metrics like perplexity potentially offer low-cost signals for Machine Translation Quality Estimation (TQE). This paper analyses perplexity in the No Language Left Behind (NLLB) multilingual model. We quantify a significant model-human perplexity gap, where the model is consistently more confident in its own, often literal, machine-generated translation than in diverse, high-quality human versions. We then demonstrate that the utility of perplexity as a TQE signal is highly context-dependent, being strongest for low-resource pairs. Finally, we present an illustrative case study where a flawed translation is refined by providing potentially useful information in a targeted prompt, simulating a knowledge-based repair. We show that as the translation’s quality and naturalness improve (a +0.15 COMET score increase), its perplexity also increases, challenging the simple assumption that lower perplexity indicates higher quality and motivating a more nuanced view of uncertainty as signalling a text’s departure from rigid translationese.

Introduction

In the field of Machine Translation (MT), evaluating output without a "gold standard" human reference remains a significant challenge, known as Translation Quality Estimation (TQE). Researchers frequently utilize a model's internal uncertainty—specifically perplexity (PPL)—as a low-cost proxy for quality. The underlying assumption is intuitive: if a model is "confused" by a sentence, that sentence is likely erroneous.

However, MT models often exhibit a bias toward "translationese"—the stiff, literal statistical patterns characteristic of machine-generated text. This research investigates the "naturalness paradox": why models are often more confident in their own mediocre translations than in fluid, high-quality human ones. By peering into the internal mechanics of the NLLB-200-3.3B model, the authors explore whether this uncertainty can be harnessed to detect errors and improve naturalness, particularly in low-resource languages.

Methodology

The study employed a "glass-box" approach to analyze the NLLB-200-3.3B model using the FLORES-200 dataset (devtest split). The researchers focused on three language pairs representing distinct resource levels: Spanish (SPA) for high-resource, Japanese (JPN) for medium-resource, and te reo Māori (MRI) for low-resource.

The methodology involved quantifying the perplexity gap between machine and human outputs, performing correlation analyses between PPL and metrics like COMET and BLEU, and conducting a token-level surprisal analysis. Finally, a "detect-and-repair" case study was performed using Llama 3 7B Instruct to fix semantic errors identified by high perplexity spikes.

The Perplexity Gap: The Narcissism of LLMs

A primary finding of the study is the consistent "perplexity gap." Across all tested languages and directions, the NLLB model assigned significantly lower perplexity to its own outputs than to professional human translations. This suggests a form of model "narcissism," where the system prefers the rigid patterns it was trained to produce over the diverse and creative structures used by humans.

Median Perplexity for Machine vs. Human text across various translation directions.

As shown in the figure above, the gap is most pronounced in directions involving low-resource languages. For English to Māori (ENG→MRI), the model was over 3.3 times more "perplexed" by human writing than by its own machine-generated text. This gap highlights the model's struggle to recognize natural, human-like prose as high-quality.

Perplexity as a Quality Signal

While the model prefers its own output, perplexity still serves as a useful quality signal within the domain of machine-generated text, especially for low-resource languages. The correlation between PPL and quality metrics (like COMET) is significantly stronger for Māori (-0.639) than for Spanish (-0.299).

Average COMET score across perplexity deciles for the MRI→ENG direction, showing quality degradation as uncertainty increases.

As illustrated in the MRI→ENG data, there is a clear downward trend: as perplexity increases (moving toward higher deciles), the average COMET score drops. This confirms that while the model is biased, extreme spikes in uncertainty still reliably point to translation failures.

The Naturalness Paradox and Repair

The study concludes with a case study that reveals the "Naturalness Paradox." By using a targeted prompt to correct a semantic error (e.g., mistranslating "spirits" as "ghosts" instead of "alcohol"), the researchers improved the translation quality by +0.15 COMET points. Interestingly, this higher-quality, more natural translation actually increased the model's perplexity, moving it closer to the human baseline. This suggests that "natural" language is inherently more surprising to a model trained on rigid patterns.

Conclusion and Future Work

Ultimately, the paper argues for a shift in perspective: instead of minimizing perplexity at all costs, researchers should treat it as a nuanced signal. High perplexity can indicate a genuine error, but it can also signal a successful departure from "translationese" toward natural prose. Future work will involve scaling these findings to larger models and validating the "naturalness" signal through direct human fluency judgments.

Summary updated

Abstract

Model-internal uncertainty metrics like perplexity potentially offer low-cost signals for Machine Translation Quality Estimation (TQE). This paper analyses perplexity in the No Language Left Behind (NLLB) multilingual model. We quantify a significant model-human perplexity gap, where the model is consistently more confident in its own, often literal, machine-generated translation than in diverse, high-quality human versions. We then demonstrate that the utility of perplexity as a TQE signal is highly context-dependent, being strongest for low-resource pairs. Finally, we present an illustrative case study where a flawed translation is refined by providing potentially useful information in a targeted prompt, simulating a knowledge-based repair. We show that as the translation’s quality and naturalness improve (a +0.15 COMET score increase), its perplexity also increases, challenging the simple assumption that lower perplexity indicates higher quality and motivating a more nuanced view of uncertainty as signalling a text’s departure from rigid translationese.
BibTeX
@misc{pistotti2025benefits,
  author = {Timothy Pistotti and Michael Witbrock and Dianne P. O’Leary and Jason Brown},
  title = {{The Benefits of Being Uncertain: Perplexity as a Signal for Naturalness in Multilingual Machine Translation}},
  year = {2025},
  doi = {10.18653/v1/2025.uncertainlp-main.7},
}
Figure