Practical Transformer-based Multilingual Text Classification

Cindy Wang, Michele BankoNAACL 2021 · DOI

TL;DR

This paper investigates the practical application of transformer-based models for multilingual text classification, moving beyond standard academic benchmarks to address real-world industry needs. They find that while multilingual models like XLM-R can be powerful, practical techniques like task/domain-adaptive pretraining and data augmentation are crucial for achieving optimal performance, even outperforming monolingual models in certain scenarios.

What Problem Does This Paper Address?

The field of Natural Language Understanding (NLU) is increasingly global, with applications needing to support thousands of languages beyond English. While powerful transformer-based models like multilingual BERT (mBERT) and XLM-ROBERTa (XLM-R) offer the promise of rapid deployment across many languages, existing academic benchmarks often focus on zero-shot cross-lingual transfer rather than supervised learning, which is more representative of industry settings. Furthermore, these benchmarks may not fully capture the nuances of data availability and task variety encountered in real-world applications. This paper aims to bridge this gap by empirically comparing various transformer-based multilingual text classification approaches in practical, industry-relevant scenarios.

Methods

The authors explore several key methods for improving multilingual text classification:

  • Pretrained Transformer Language Models: They compare both monolingual models (e.g., RoBERTa for English, German BERT for German) and massively multilingual models (XLM-R).
  • Adaptive Pretraining:
    • Domain-Adaptive Pretraining (DAPT): Further pretraining on a large corpus of unlabeled domain-specific text.
  • Task-Adaptive Pretraining (TAPT): Further pretraining on the training data of the specific downstream task. Both DAPT and TAPT use the Masked Language Modeling (MLM) objective, similar to the original pretraining of BERT and RoBERTa.

  • Supervised Fine-Tuning Settings:

    • Mono-target: Fine-tuning a monolingual LM on data for each target language (resulting in N models for N languages).
  • Multi-target: Fine-tuning XLM-R on data for each target language (resulting in N models).
  • Multi-all: Fine-tuning XLM-R on the concatenation of all training data (resulting in one model).
  • Data Augmentation: Employing machine translation to translate existing training data into other languages, thereby increasing the amount of labeled data available.

Datasets

The paper uses two primary datasets for evaluation:

  • Cross-Lingual Sentiment dataset (CLS): This dataset consists of Amazon product reviews in four languages (English, German, French, Japanese) across three product categories. It includes labeled training and test sets, as well as unlabeled data for adaptive pretraining.
  • HATEVAL dataset: This dataset contains tweets in English and Spanish annotated for hate speech targeting women and immigrants. The authors note and address inconsistencies in the original English test set by manually relabeling a portion of it, providing a more reliable benchmark. They also collected unlabeled Twitter data for DAPT.

Key Insights and Ideas

  • Practical Modifications Matter: Task- and domain-adaptive pretraining (TAPT/DAPT) and data augmentation via machine translation consistently improve performance across tasks and languages, often without needing additional human-labeled data.
  • Multilingual Models Aren't Always Superior (But Often Are): For high-resource languages with extensive monolingual pretraining corpora (like English and French), monolingual models can sometimes outperform multilingual ones. However, for languages with more limited pretraining data (like German and Japanese), massively multilingual models like XLM-R often perform better.
  • Task Nuance is Key: The performance differences between monolingual and multilingual models can be attributed to the nature of the task. Sentiment analysis, being a more well-defined linguistic facet, might benefit more from specialized lexical information captured by monolingual models. Hate speech detection, which is more nuanced and context-dependent, may leverage the broader knowledge of multilingual models.
  • Zero-Shot is Useful, But Supervised is Better: While zero-shot learning with multilingual models is competitive with prior baselines, supervised fine-tuning with even a small amount of target language labels yields significantly better results, making it more practical for many industry applications.
  • Trade-offs in Deployment: The choice between many monolingual models and one large multilingual model involves a trade-off between deployment complexity and model size/memory usage.

Key Results

  • TAPT, DAPT, and data augmentation improved macro-F1 scores by 0.6-1.5% for CLS and 0.3-4.3% for HATEVAL.
  • For CLS, monolingual models (mono-target) performed best for English and French. For German and Japanese, XLM-R models (multi-target and multi-all) showed superior performance.
  • On HATEVAL, XLM-R models generally outperformed those fine-tuned from monolingual LMs, even for English.
  • The relabeled HATEVAL English test set showed significantly higher scores than the original, highlighting the importance of data quality.
  • Inference benchmarks showed that while monolingual BERT models are similar in speed, XLM-R's memory usage becomes prohibitive at batch sizes larger than 32, though it remains comparable at small batch sizes (e.g., 1).

Limitations and Future Work

  • Limitations: The paper acknowledges that XLM-R can suffer from "curse of multilinguality" (performance degradation with too many languages) in some cases. The inference benchmarks highlight memory constraints for XLM-R at larger batch sizes.
  • Future Work: The authors suggest exploring cross-domain and cross-task transfer, low-resource and few-shot learning, and practical alternatives to large multilingual models like distillation. They also emphasize the need for more realistic benchmarks that account for factors like domain mismatch and limited labeled data.

Summary updated

BibTeX
@inproceedings{wang2021practical,
  author = {Cindy Wang and Michele Banko},
  title = {{Practical Transformer-based Multilingual Text Classification}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2021},
  doi = {10.18653/v1/2021.naacl-industry.16},
}
Figure