MLCL: A Framework for Reducing Language Imbalance in Sino-Tibetan Languages through Adapter Structures

JiaJun Fang, Wentao Huang, Aimin Yang, Dong Zhou, Nankai LinACML 2024 · OpenReview

TL;DR

MLCL (MultiLingual Contrastive Learning) is a novel framework designed to fix the "language imbalance" problem where AI models perform significantly better on high-resource languages than on low-resource ones. By leveraging phonetic similarities within the Sino-Tibetan language family and combining text with synthetic speech data, the researchers achieved more equitable and accurate performance across Chinese, Tibetan, and Burmese.

What Problem Does This Paper Address?

Multilingual pre-trained models like mBERT are the workhorses of modern NLP, but they have a "rich-get-richer" problem. Because they are trained on massive amounts of data from high-resource languages (like English or Chinese), their performance on low-resource languages (like Tibetan or Burmese) often lags far behind.

While researchers have tried to bridge this gap using cross-lingual text or image data, they have largely ignored audio. Languages within the same family—such as the Sino-Tibetan family—often share deep phonetic roots. The authors argue that ignoring these auditory nuances leaves a massive amount of "linguistic signal" on the table, resulting in an incomplete solution to language imbalance.

Methods

The authors introduce the MultiLingual Contrastive Learning (MLCL) framework, which treats text classification as a multi-modal task.

  1. Feature Extraction: They use mBERT for textual features and wav2vec 2.0 for audio features. To make the audio model more efficient, they incorporate "adapters" (MetaAdapter and SimAdapter)—small, tunable layers that allow the model to adapt to new languages without retraining the entire massive wav2vec backbone.
  2. Two-Pronged Contrastive Learning:
    • Language-based Contrastive Learning (LCL): This task forces the model to minimize the distance between different languages within the same modality (e.g., making the "concept" of a sentence in Chinese audio look similar to the same concept in Tibetan audio).
  3. Modality-based Contrastive Learning (MCL): This aligns text and audio features within the same language, ensuring the model understands that the written word and the spoken word represent the same thing.
  4. Fusion & Classification: The features are fused using CM-BERT (Cross-Modal BERT) to capture interactions between text and sound before making a final classification.
  5. A New Metric: To measure "fairness," the authors borrowed the Gini coefficient from economics. In this context, a lower Gini coefficient means the model’s accuracy is more evenly distributed across all languages, rather than being "hoarded" by the high-resource ones.

Datasets

The study focused on three Sino-Tibetan languages: Chinese, Tibetan, and Burmese. Because parallel audio-text datasets for these languages are scarce, the authors created two synthetic datasets:

  • Flores200 (Domain Classification): They extracted parallel text and used Text-to-Speech (TTS) tools (gTTS and MaryTTS) to generate 20.22 hours of audio.
  • mms (Sentiment Analysis): They took Chinese sentiment data, translated it into Burmese and Tibetan using GPT-4, and then generated 60.18 hours of parallel audio via TTS.

Key Insights and Ideas

  • Phonetics as a Bridge: The core "aha moment" is that phonetic similarity is a powerful anchor. Even if the scripts (writing systems) of two languages look completely different, their sounds often carry shared evolutionary traits that a model can exploit.
  • The Fairness Trade-off: Usually, when you try to improve a model's performance on a low-resource language, the performance on the high-resource language drops (a "see-saw" effect). MLCL manages to raise the floor for Tibetan and Burmese while keeping the ceiling high for Chinese.
  • Adapters are Essential: The ablation studies showed that removing the adapter structure caused the single biggest drop in performance. This suggests that specialized "tuning" for audio is more important than just having the audio data itself.

Key Results

The MLCL framework outperformed all baselines (including standard mBERT and adversarial training methods like PGD and FGM) in both overall accuracy and language balance.

  • On Flores200: MLCL achieved a Mean Accuracy (M) of 0.6420 and the lowest Gini coefficient (G) of 0.1245. For comparison, the best baseline (mBERT-Combined) had a Gini of 0.1427, indicating higher inequality between languages.
  • On mms: MLCL reached an M of 0.5671 and a G of 0.1477, again proving to be the most balanced and accurate model.
  • Low-Resource Boost: In Tibetan and Burmese, MLCL saw accuracy improvements of roughly 1-3% over the strongest baselines, while maintaining competitive results in Chinese.

Limitations and Future Work

  • Synthetic vs. Real Audio: The use of TTS-generated audio is a clever workaround, but synthetic speech lacks the noise, accents, and emotional prosody of real human speech. The authors acknowledge the need for real-world data.
  • Limited Language Scope: The study only looked at three languages. The Sino-Tibetan family is vast, and future work needs to see if these phonetic links hold up across dozens of dialects.
  • Expanding Modalities: The authors suggest that adding images (tri-modal learning) could further reduce imbalance, as visual concepts are often universal across cultures.
  • Scaling Up: The current work uses base-sized models; testing these contrastive tasks on Large Language Models (LLMs) is a logical next step.

Summary updated

Abstract

Multilingual pre-trained models have been widely applied in natural language processing (NLP) tasks, including text classification. However, due to the varying amounts of language resources, these models exhibit performance imbalance across different languages, a phenomenon known as language imbalance. Existing research on mitigating language imbalance primarily harnesses text and image data, neglecting the auditory aspects of languages. This neglect results in an incomplete solution to language imbalance, as it fails to exploit the rich linguistic nuances conveyed through speech. To address these issues, this paper introduces a novel framework called MultiLingual Contrastive Learning (MLCL) to reduce language imbalance. By incorporating concepts from comparative linguistics into neural networks, we utilize the phonetic similarities among languages within the Sino-Tibetan family to tackle the problem of language imbalance in multilingual pre-trained models. To evaluate our method's effectiveness, we conducted tests using two synthetic datasets derived from the Flores200 and mms datasets across various models. The experimental results show that, in terms of language imbalance metrics, our model surpasses all baseline models.
BibTeX
@inproceedings{fang2024mlcl,
  author = {JiaJun Fang and Wentao Huang and Aimin Yang and Dong Zhou and Nankai Lin},
  title = {{MLCL: A Framework for Reducing Language Imbalance in Sino-Tibetan Languages through Adapter Structures}},
  booktitle = {ACML},
  year = {2024},
  url = {https://openreview.net/forum?id=s9yXKcsKbP},
}
Figure