From Shortcuts to Balance: Attribution Analysis of Speech-Text Feature Utilization in Distinguishing Original from Machine-Translated Texts

Yongjian Chen, Antonio ToralEMNLP 2025

TL;DR

This paper investigates how AI models distinguish between human-written and machine-translated text, revealing that models often rely on superficial "shortcuts" like named entities (NEs) rather than genuine linguistic cues of translationese. By analyzing both text and speech features, the authors demonstrate that combining these modalities leads to more balanced feature utilization, reducing reliance on NEs and improving robustness.

What Problem Does This Paper Address?

The increasing prevalence of machine translation (MT) necessitates reliable methods for detecting machine-generated text. While AI models have become adept at this task, research has shown they often exploit superficial patterns, such as the presence of named entities (NEs), rather than truly understanding the linguistic nuances that differentiate human-authored text from MT output. This "Clever Hans" behavior, where models learn spurious correlations (e.g., associating specific locations with translated texts), makes them brittle and less reliable. Previous attempts to mitigate this, like masking NEs, reduced performance. This paper aims to understand how integrating speech features with text can help models achieve a more balanced and robust understanding of translationese, moving beyond simple shortcuts.

Methods

The core of this research lies in a systematic attribution analysis framework. Using Integrated Gradients (IG), a technique for explaining model predictions by attributing importance to input features, the researchers analyze how different parts of the input (text and speech) contribute to the model's decision.

Key methodological aspects include:

  • Unimodal and Bimodal Analysis: They analyze text-only, speech-only, and combined text-speech models.
  • Attribution Computation: IG is applied to text embeddings and raw audio features.
  • Attribution Aggregation: To move beyond token-level analysis, they aggregate attribution scores at both word and entity (NE or non-NE span) levels. This allows for a finer-grained understanding of what specific linguistic units the model focuses on.
  • Normalization Strategies: Various normalization methods (mean, L1, signed-max) are used to quantify attribution patterns.
  • Cross-Modal Comparison: The framework is designed to specifically compare how attribution patterns change when moving from unimodal to bimodal settings, and how different training conditions (e.g., NE masking, frozen vs. unfrozen speech parameters) affect these patterns.

Datasets

The study utilizes data from the WMT news task for English and Chinese. German source texts are translated into English and Chinese using Google Translate. The training data spans from 2014-2017 for English and 2017-2019 for Chinese, with 2018 and 2020 used for testing. Speech data is synthesized using Microsoft Azure TTS API's male voices.

Key Insights and Ideas

  • NEs as Shortcuts: Text-based models heavily rely on named entities (NEs) as a shortcut for identifying machine-translated text, especially in English. This reliance is more pronounced at the entity level than at the word level.
  • Speech Features Offer Balance: Integrating speech features helps to mitigate the text modality's over-reliance on NEs. This bimodal integration leads to more balanced feature utilization.
  • Frozen Speech Parameters are Key: For bimodal fusion, using speech models with frozen parameters (mimicking a two-stage approach) is preferred. Unfrozen speech models, while achieving higher accuracy, show a stronger dependence on NEs, which is undesirable.
  • Cross-Lingual Differences: The impact of NE masking and bimodal integration shows variations between English and Chinese, suggesting language-specific nuances in how translationese is manifested and detected. English shows a more pronounced shift in attribution patterns.
  • Complementary Modalities: The bimodal approach creates a complementary effect. In English, text's reduced NE reliance is balanced by speech features shifting focus, leading to more distributed attribution. In Chinese, speech features maintain their non-NE emphasis, indicating adaptive balancing strategies.

Key Results

  • Accuracy: While the streamlined bimodal approach shows a slight accuracy decline compared to previous two-stage methods, it still maintains robust performance. Bimodal classifiers consistently outperform NE-masked text-based counterparts.
  • Attribution Shifts:
    • Text: NE masking significantly reduces the reliance on NEs in text classifiers, with a more pronounced effect in English (Δ 1.32) than Chinese (Δ 0.25).
  • Speech: Unfrozen speech models attribute more importance to NEs than frozen ones, raising concerns about their reliance on shortcuts.
  • Bimodal: The transition from unimodal to bimodal settings leads to reduced NE reliance in text features. For speech features, the effect is more nuanced, with English showing an increase in NE attribution in bimodal settings but still maintaining a smaller shift compared to the text modality's changes.

Limitations and Future Work

The authors acknowledge several limitations and suggest future research directions:

  • NMT Architectures: The analysis is based on one widely-used translation system. Investigating other NMT architectures could reveal variations in feature utilization.
  • Speech Synthesis: The use of specific TTS voices (male) for speech synthesis might not fully represent the diversity of human speech. Exploring a spectrum of voices with varied accents, genders, and emotions could offer deeper insights.
  • Translation Directions: Expanding the analysis to more diverse language pairs beyond German-to-English and German-to-Chinese could enhance understanding of attribution patterns across different typological relationships.
  • Alternative Data Sources: Utilizing naturally occurring speech-text pairs, such as from audiobooks or dubbed movies, could provide more realistic data for analysis.

Summary updated

BibTeX
@inproceedings{chen2025from,
  author = {Yongjian Chen and Antonio Toral},
  title = {{From Shortcuts to Balance: Attribution Analysis of Speech-Text Feature Utilization in Distinguishing Original from Machine-Translated Texts}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure