Unsupervised Outlier Detection for Language-Independent Text Quality Filtering

Jón Daðason, Hrafn LoftssonCOLING 2024

TL;DR

This paper explores unsupervised methods for filtering low-quality text from large web-crawled corpora, a crucial step for training effective language models. They demonstrate that simple outlier detection algorithms, like Gaussian Mixture Models, can achieve performance comparable to meticulously tuned rule-based systems, without requiring language expertise or labeled data.

What Problem Does This Paper Address?

Training large language models (LLMs) relies heavily on vast amounts of text data scraped from the web. However, this data is often noisy, containing a significant amount of low-quality content such as machine translations, errors, or incoherent text. This noise can degrade the performance of LLMs. Traditionally, filtering this noisy text involves either complex rule-based systems that require careful tuning and linguistic expertise, or supervised classifiers that demand manually labeled datasets. The challenge is to find an efficient, language-independent, and unsupervised method to effectively filter this noisy text.

Methods

The authors investigate two main approaches:

  • Rule-based Filtering: They analyze the effectiveness of commonly used document-level filtering rules, such as those found in the ROOTS and MassiveWeb corpora. These rules assess metrics like perplexity, word length, stop word ratios, and character repetition. They use a grid search with cross-validation on a manually labeled Icelandic dataset (TQ-IS) to find optimal rule combinations and thresholds.
  • Unsupervised Outlier Detection: They propose reframing text quality filtering as an outlier detection problem. They evaluate three unsupervised algorithms:
  • Gaussian Mixture Models (GMMs): A probabilistic model that assumes data is composed of a mixture of Gaussian distributions, used here for clustering.
  • One-Class Support Vector Machines (OCSVMs): An algorithm that learns a boundary around the "normal" data points, identifying those outside as outliers.
  • Isolation Forests: An ensemble method that isolates outliers by randomly partitioning data.

These unsupervised methods are applied using features extracted from the documents, similar to those used in the rule-based approach.

Datasets

  • TQ-IS (Icelandic Text Quality Dataset): A manually labeled dataset of 2,000 Icelandic documents, serving as the primary benchmark for evaluating filtering methods. Documents are labeled as either high-quality or low-quality.
  • mC4 Corpus: A large, multilingual web-crawled corpus used for training and evaluating the unsupervised models on Icelandic, Estonian, and Basque subsets.
  • Curated Corpora: Icelandic Gigaword Corpus (IGC), Estonian National Corpus (ENC), and EusCrawl were used to train language models for feature extraction (e.g., perplexity) and as training data for the unsupervised models.

Key Insights and Ideas

  • Perplexity is King (for rules): Among the rule-based metrics, perplexity (a measure of how well a language model predicts a text) emerged as the single most effective feature for distinguishing high-quality from low-quality text.
  • Simplicity Wins (for rules): Surprisingly, a small subset of rules (just five out of thirteen tested) was sufficient to achieve near-optimal performance in the rule-based approach. This suggests that many rules might be redundant or even detrimental when combined.
  • Outliers are Low-Quality: Visualizing document features (like perplexity and stop word ratio) on the TQ-IS dataset revealed that high-quality documents form a dense cluster, while low-quality documents appear as sparse outliers. This observation strongly supports the outlier detection approach.
  • Unsupervised is Competitive: Unsupervised methods, particularly GMMs, achieved performance comparable to the best rule-based approaches, without the need for manual labeling or language-specific tuning.
  • Language Independence: The success of unsupervised methods on Icelandic suggests their potential applicability to other languages, including those with fewer resources, as they don't rely on language-specific linguistic knowledge.

Key Results

  • Rule-based: An optimal rule-based approach using five rules achieved an F₁ score of 98.2% on the TQ-IS dataset. Even without resource-intensive rules, an F₁ score of 97.43% was attainable.
  • Unsupervised:
    • Gaussian Mixture Models (GMMs) achieved the highest F₁ score of 98.32% on TQ-IS, outperforming both OCSVM and Isolation Forests.
  • OCSVM achieved 96.40%, and Isolation Forest achieved 97.52%.
  • Training Set Size: The performance of unsupervised models plateaued significantly after training on approximately 5,000 documents, indicating efficiency and effectiveness even with limited data.

Limitations and Future Work

  • Limited Labeled Data: The primary limitation is the lack of document-level text quality datasets for languages other than Icelandic. This prevents empirical validation of unsupervised methods on a wider range of languages. However, visualizations on Estonian and Basque mC4 subsets suggest similar clustering patterns.
  • Future Work: The authors plan to investigate how different categories of low-quality text impact downstream LLM performance. This could lead to better text quality datasets and improved filtering strategies. They also aim to construct text quality datasets for other languages.

Summary updated

BibTeX
@inproceedings{daason2024unsupervised,
  author = {Jón Daðason and Hrafn Loftsson},
  title = {{Unsupervised Outlier Detection for Language-Independent Text Quality Filtering}},
  booktitle = {International Conference on Computational Linguistics},
  year = {2024},
}
Figure