uniblock: Scoring and Filtering Corpus with Unicode Block Information

Yingbo Gao, Weiyue Wang, Hermann Ney

TL;DR

This paper introduces "uniblock," a novel, unsupervised method for filtering noisy sentences from text corpora using Unicode block information. By modeling character distributions with a Gaussian Mixture Model, uniblock effectively identifies and removes sentences with unusual character compositions, leading to improved performance in downstream NLP tasks like sentiment analysis, language modeling, and machine translation.

What Problem Does This Paper Address?

In Natural Language Processing (NLP), cleaning text data is a crucial preprocessing step. A common heuristic is to remove sentences containing "illegal characters." However, defining what constitutes an "illegal character" is highly context-dependent, varying with task, language, and domain. This often leads to tedious, repetitive manual rule-scripting. For instance, modern Chinese text can legitimately include a wide array of characters, from traditional and simplified Chinese to emojis, mathematical symbols, and Latin characters. Manually creating robust rules to distinguish legitimate noise from truly problematic content is a significant challenge.

Methods

The core of uniblock lies in its statistical approach to identify "noisy" sentences without relying on hand-crafted rules.

  1. Feature Engineering: For each sentence, a feature vector is generated based on the Unicode block information of its characters. Specifically, the method counts the occurrences of characters within each of the 300 Unicode blocks. These counts are then normalized to form a probability distribution over blocks. To handle sparsity (many blocks having zero counts), dimensionality reduction is applied by only considering blocks with non-zero counts in the training data.
  2. Probabilistic Modeling: A Gaussian Mixture Model (GMM) is trained on a "clean" corpus (e.g., a development set). The GMM assumes that the feature vectors of clean sentences are generated from a mixture of Gaussian distributions. Variational inference is used to estimate the model parameters, which has the advantage of automatically determining the number of mixture components.
  3. Scoring and Filtering: Once trained, the GMM can assign a score to any given sentence's feature vector. Sentences with lower scores are considered more likely to contain unusual or "illegal" characters. The paper proposes two filtering strategies:
  4. Absolute Thresholding: Removing sentences below a fixed score.
  5. Relative Thresholding: Removing a fixed percentage of the lowest-scoring sentences. The method also supports filtering parallel corpora by combining scores from paired sentences (e.g., using minimum, maximum, or average).

Datasets

The paper evaluates uniblock on several datasets:

  • Sentiment Analysis: Stanford Twitter Sentiment (STS) corpus.
  • Language Modeling: WMT19 Chinese-English (zh-en) and Russian-English (ru-en) parallel corpora (training data), newsdev2017 (zh-en development), and newstest2016 (ru-en development). Test data included newstest2017 and newstest2018.
  • Machine Translation: The same WMT19 parallel corpora used for LM experiments, for training translation systems in four directions (zh-en, en-zh, ru-en, en-ru).

No new benchmarks were introduced; existing datasets were used to demonstrate the effectiveness of the filtering method.

Key Insights and Ideas

  • Data-Driven Rules: The paper successfully demonstrates that instead of manually defining "illegal" characters, statistical models can learn to identify them from data, offering a more adaptable and less labor-intensive approach.
  • Unicode Blocks as Powerful Features: Leveraging Unicode block information proves to be a surprisingly effective way to capture sentence-level character composition, even for languages with complex scripts.
  • GMM for Noise Detection: A Gaussian Mixture Model, trained on clean data, can effectively model the distribution of "normal" character compositions, allowing for the identification of outliers.
  • Simplicity and Effectiveness: The uniblock method is presented as a simple yet effective statistical heuristic that consistently improves performance across diverse NLP tasks.

Key Results

  • Sentiment Analysis: Uniblock improved test accuracy on the Stanford Twitter Sentiment corpus by 0.6% over a baseline with minimal preprocessing.
  • Language Modeling: Consistent improvements were observed for zh, ru, and en under zh-en LM tasks when using uniblock-filtered corpora, with perplexity scores decreasing. For example, for the zh-en task, 10% filtering resulted in a perplexity of 135.0 compared to the baseline's 145.7.
  • Machine Translation: Uniblock filtering led to consistent improvements in case-sensitive BLEU scores across four translation directions (zh-en, en-zh, ru-en, en-ru), with gains up to 1.1 BLEU points.

Limitations and Future Work

  • Indistinguishable Blocks: The paper acknowledges that uniblock might fail if "clean" and "dirty" sentences share the exact same set of Unicode blocks after normalization. In such cases, designing alternative features or using different modeling approaches would be necessary.
  • Future Directions: The authors suggest several avenues for future work:
  • Exploring other dimensionality reduction techniques.
  • Investigating alternatives to GMM for modeling.
  • Incorporating additional heuristics into the feature vector, such as sentence length and punctuation marks.
  • Extending the method to handle more nuanced definitions of "noise" beyond just character composition.

Summary updated

Abstract

The preprocessing pipelines in Natural Language Processing usually involve a step of removing sentences consisted of illegal characters. The definition of illegal characters and the specific removal strategy depend on the task, language, domain, etc, which often lead to tiresome and repetitive scripting of rules. In this paper, we introduce a simple statistical method, uniblock, to overcome this problem. For each sentence, uniblock generates a fixed-size feature vector using Unicode block information of the characters. A Gaussian mixture model is then estimated on some clean corpus using variational inference. The learned model can then be used to score sentences and filter corpus. We present experimental results on Sentiment Analysis, Language Modeling and Machine Translation, and show the simplicity and effectiveness of our method.
BibTeX
@inproceedings{gao2019uniblock,
  author = {Yingbo Gao and Weiyue Wang and Hermann Ney},
  title = {{uniblock: Scoring and Filtering Corpus with Unicode Block Information}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2019},
  doi = {10.18653/v1/d19-1133},
}
Figure