TL;DR
Can machine translation make AI fairer? This research demonstrates that round-trip translation (RTT)—the process of translating text into a foreign language and back again—acts as a powerful "text normalizer" that reduces demographic bias in sentiment analysis. By stripping away group-specific linguistic quirks, RTT at test time can reduce fairness gaps between different age and gender groups by up to 47%.
The Challenge of Demographic Bias
Machine learning models for sentiment analysis often perform inconsistently across different demographic groups. For example, a classifier might be more accurate at identifying sentiment in reviews written by young men than in those written by older women. This disparity occurs because models inadvertently learn "protected attributes" (like age or gender) through the specific linguistic styles and vocabularies used by different groups, leading to unethical performance gaps.
While various mitigation strategies exist, most require access to protected demographic attributes during the training phase—data that is rarely available in real-world applications. This study investigates whether RTT can serve as an unsupervised method to "wash away" these demographic markers by forcing text into a more standardized or normalized form.
Methodology
The researchers implemented a multi-stage pipeline to test the efficacy of RTT:
- Round-Trip Translation (RTT): Danish reviews were translated into English and then back into Danish using Google Cloud Translation.
- Normalization Check: The team used KL-divergence (KLD) to measure how much the word distributions of different demographic groups converged after RTT. They observed that RTT significantly reduced group-level linguistic differences and decreased the number of unique words by 36%.
- Classification Models: Nine different architectures were tested using two types of text embeddings: LASER (a multilingual sentence encoder) and Danish BERT (a monolingual transformer model).
- Experimental Scenarios: The study compared three distinct setups:
- Baseline: Training and testing on original reviews.
- Test-time RTT: Training on original reviews, but applying RTT to the test data.
- Full RTT: Training and testing on RTT-processed data.
Dataset Overview
The study utilized the Danish Trustpilot Corpus, comprising 149,240 reviews. These reviews include self-reported gender and age metadata. For the analysis, sentiment scores (1–5 stars) were binarized into "negative" (1–3) and "positive" (4–5). The test set was categorized into six groups based on gender and three age brackets (≤42, 43–55, and >55).
Key Insights: Translation as a Filter
RTT acts as a "style transfer" mechanism, replacing idiosyncratic or slang-heavy expressions with more common, normalized equivalents. For instance, a phrase like "Sally is a whiz at math" might be normalized to "Sally is a math expert."
Because RTT does not require labels for gender or age, it is highly scalable to tasks where demographic data is missing. However, the study found a divide between BERT and LASER models. While RTT consistently helped at test time, retraining BERT on RTT data occasionally widened the fairness gap, suggesting that transformer-based representations might be sensitive to specific biases introduced by the translation process itself.
Experimental Results
The primary metric for fairness was the Fairness Gap, defined as the difference in F1-score between the best-performing and worst-performing demographic groups.
As shown in the results, applying RTT consistently reduced the fairness gap. For example, in the BERT-KNN model, the gap dropped from 0.200 in the baseline to 0.107 in the full RTT scenario—a reduction of approximately 47%.
However, this improvement comes with a trade-off: overall accuracy generally decreased slightly. The authors describe this as a fundamental tension between achieving fairness and maintaining raw predictive performance. In most cases, RTT improved fairness by making the model slightly less accurate for everyone, rather than specifically improving performance for disadvantaged groups (with LASER models being a notable exception).
Limitations and Future Directions
While RTT removes human bias, it may introduce "machine bias" inherent in translation algorithms, such as the introduction of gendered pronouns where none existed. Furthermore, the performance cost may be prohibitive for certain high-stakes applications.
Future research could explore the impact of different "pivot" languages (e.g., Danish to Chinese) to see if greater linguistic distance enhances the normalization effect, or investigate methods to mitigate the absolute performance drop associated with RTT.
Summary updated