Introduction
State-of-the-art machine translation (MT) systems and evaluation metrics are often assumed to be robust to minor textual variations. However, this research demonstrates that they are surprisingly fragile when it comes to sentence-final punctuation. A single added or deleted full stop can trigger hallucinations, flip meanings, or cause massive drops in evaluation scores. This "punctuation sensitivity" calls into question the reliability of these systems for real-world, noisy data where punctuation is often inconsistent.
Problem Statement
While researchers have studied how MT handles "heavy" noise like spelling errors or dialects, the impact of basic punctuation is often overlooked. We intuitively expect that adding a period or an exclamation point shouldn't fundamentally change the core meaning of a sentence. This paper addresses this gap by investigating whether commercial MT systems (Google Translate, DeepL, Microsoft) and standard metrics (BLEU, COMET, etc.) are robust to the insertion or deletion of sentence-final punctuation.
Methodology
The researchers employed a two-pronged approach to evaluate both the "graders" and the "translators":
- Meta-Evaluation of Metrics: The authors tested if standard metrics (BLEU, chrF, COMET, BERTScore, BLEURT) unfairly penalize tiny punctuation changes. They created "perturbed" versions of sentences by adding or removing punctuation and checked the resulting score drops.
- System Perturbation Testing: Three major commercial MT systems—Google, DeepL, and Microsoft—were evaluated across three language pairs: German (De), Japanese (Ja), and Ukrainian (Uk) into English (En).
- Punctuation Reset: To ensure a fair comparison, the researchers "reset" the punctuation in the MT output (e.g., removing a period the model added because the input had one) before evaluation. This allowed them to see if the content of the translation actually changed.
- Manual Analysis: A qualitative review was performed to categorize error types, such as reordering, tense changes, or hallucinations.
Datasets
The study utilized high-quality benchmarks including FLORES101 and WMT 2020-2022. These sets were filtered to create specific splits based on original punctuation (e.g., a subset of sentences that originally had no final punctuation for the "insertion" test).
Key Insights
- Metric Hypersensitivity: Model-based metrics like COMET and BERTScore often penalize a mismatched punctuation mark more severely than the insertion of a random character. This suggests these metrics may prioritize surface-level matching over semantic consistency.
- The "Headline" Bias: Models appear to associate a lack of punctuation with news headlines. Deleting a full stop often causes the model to shift to a "headline style," producing shorter, punchier, but less accurate translations.
- Hallucinations and Meaning Flips: In several cases, adding a punctuation mark caused the model to hallucinate entirely new words or even negations (e.g., adding "not" where it didn't exist), completely reversing the meaning of the source text.
- Language-Specific Robustness: Japanese and Ukrainian systems showed much higher inconsistency (up to 35%) compared to German. This suggests that the resource level and script of a language impact how robustly models learn punctuation patterns.
Key Results
The quantitative results for commercial systems show significant performance drops and high inconsistency rates across all tested languages.
Qualitative analysis further illustrates the severity of these failures. For example, adding an exclamation mark changed "lets Tesla shares rise" to "makes Tesla shares rise," while adding a question mark changed a statement about "advice" into a polite request "Please give me some advice."
In meta-evaluation, the impact was equally stark: adding a question mark to a German sentence caused a 24.9 point drop in COMET scores, even though the text was otherwise identical. Generally, deleting punctuation was found to be more disruptive than inserting it.
Limitations and Future Work
The authors acknowledge that their test subsets were sometimes small due to the specific punctuation requirements. Furthermore, the study was English-centric, focusing only on translations into English. Future research is needed to investigate these sensitivities in morphologically rich target languages and to explore the internal mechanics of Transformer models to explain why a single terminal token can so drastically reweight an entire sequence.
Summary updated