On the Utility of Pretraining Language Models on Synthetic Data

Alcides Alcoba Inciarte, Sang Yun Kwon, El Moatez Billah Nagoudi, Muhammad Abdul-MageedACL 2024

TL;DR

This paper explores the effectiveness of pre-training Arabic language models on synthetic data generated from various sources (OCR, ASR, MT, text generation) compared to traditional "gold" data. The findings demonstrate that synthetic data can achieve comparable, and sometimes superior, performance on Arabic NLU tasks, offering a promising avenue for low-resource settings.

What Problem Does This Paper Address?

Developing high-performing Natural Language Processing (NLP) models, especially for languages like Arabic, often requires massive amounts of labeled data. This reliance on abundant data presents a significant challenge in low-resource settings where acquiring and annotating such datasets is costly and time-consuming. The paper addresses this by investigating whether synthetic data, generated through various automated methods, can serve as a viable alternative or supplement to traditional, manually curated datasets for pre-training Arabic language models.

Methods

The researchers pre-trained language models using three distinct architectures: encoder-only (BERT Base), encoder-decoder (T5), and decoder-only (GPT-2). They employed four different methods to generate synthetic training data:

  • Optical Character Recognition (OCR): Extracting text from scanned documents (specifically, 3,000 books from the Arabic public library Hindawi).
  • Automatic Speech Recognition (ASR): Transcribing audio data from various Arabic speech datasets.
  • Machine Translation (MT): Translating English Wikipedia articles into Arabic using Meta's No Language Left Behind model.
  • Text Generation (TG): Using an Arabic GPT model (Jasmine) to complete sentences and generate new text based on existing content.

These models were then evaluated on the ORCA benchmark, a comprehensive Arabic NLU benchmark comprising 60 datasets across seven task clusters.

Datasets

  • Training Data:
    • Gold Data: Raw text extracted from Arabic Wikipedia articles.
  • Synthetic Data: * MT Data: Translated English Wikipedia articles.
  • OCR Data: Text extracted from 3,000 Arabic books via OCR.
  • ASR Data: Transcribed audio from various Arabic speech datasets (MSA and dialects).
  • TG Data: Synthetically generated text using the Jasmine Arabic GPT model.
  • Evaluation Benchmark:
    • ORCA Benchmark: A diverse NLU benchmark for Arabic, covering Modern Standard Arabic (MSA) and Dialectal Arabic (DA), with 60 publicly available datasets segmented into seven task clusters (Sentence Classification, Text Classification, Structured Prediction, Semantic Text Similarity, Natural Language Inference, Question-Answering, and Word Sense Disambiguation).

Key Insights and Ideas

  • Synthetic Data's Power: The study strongly suggests that synthetic data can be a powerful substitute for or complement to gold-standard data, especially in low-resource scenarios.
  • Diversity is Key: The ASR dataset, which included a mix of MSA and various Arabic dialects, consistently led to the best performance for the BERTBase model, highlighting the importance of diverse linguistic coverage.
  • Architecture Matters: Different model architectures (BERT, GPT-2, T5) showed varying sensitivities to different synthetic data sources, indicating that the choice of architecture and data source should be carefully considered.
  • GPT-2's Adaptability: The GPT-2 model, being the largest, showed continuous improvement with combined synthetic datasets and generally outperformed BERT Base and T5, suggesting its strong capacity for learning from limited data.
  • Limitations of Single Sources: Relying on a single, non-diverse data source (like solely Wikipedia) can limit model pre-training effectiveness.

Key Results

  • ASR Data Dominance (BERTBase): The BERTBase model pre-trained on ASR data achieved the highest ORCAscore, outperforming the gold dataset by approximately 5 F1 points.
  • GPT-2 Superiority: The GPT-2 model achieved the highest ORCAscore across all datasets compared to other architectures, with the gold-data trained model performing best overall but struggling in STS tasks.
  • T5 Performance: The T5 model trained on ASR data also achieved the highest ORCAscore, surpassing the gold dataset, particularly excelling in abusive, emotion, and emotion-reg tasks.
  • Comparable Performance: In many tasks, models trained on synthetic data achieved performance comparable to, or even surpassing, those trained on gold data. For instance, a GPT-2 model trained on a combined synthetic dataset outperformed the ARBERTV2 baseline.
  • TG Data's Novelty: While MT data generally underperformed, TG data showed improved results due to its novelty and the robust language model used in its creation.

Limitations and Future Work

  • Dialectal Data Scarcity: Only the ASR dataset contained significant dialectal Arabic (DA) data, while other datasets were primarily Modern Standard Arabic (MSA). This imbalance might have skewed results, especially for DA-related tasks.
  • Model Size Discrepancy: The study acknowledges that the varying model sizes across architectures (BERT Base, GPT-2, T5) could affect direct performance comparisons. Future work could focus on similarly sized models.
  • NLU Focus: The study was primarily focused on NLU tasks. The findings might not generalize to other NLP tasks like Natural Language Generation (NLG).
  • Dataset Biases: Datasets like OCR, sourced from academic texts, may lack informal language, potentially limiting performance on tasks involving colloquialisms or slang. Future work could explore mitigating these domain-specific biases.
  • Future Work: The authors suggest exploring more diverse data combinations and investigating the impact of model size for fairer comparisons. They also propose extending the research to NLG tasks.

Summary updated

BibTeX
@inproceedings{inciarte2024utility,
  author = {Alcides Alcoba Inciarte and Sang Yun Kwon and El Moatez Billah Nagoudi and Muhammad Abdul-Mageed},
  title = {{On the Utility of Pretraining Language Models on Synthetic Data}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
}
Figure