Data Annealing for Informal Language Understanding Tasks

Jing Gu, Zhou Yu

Introduction

Large language models like BERT have demonstrated exceptional performance in understanding formal language, such as Wikipedia articles and news reports. However, their efficacy often declines significantly when applied to "informal" language—the slang-heavy, typo-ridden, and grammatically idiosyncratic text found on social media platforms like Twitter. This performance gap stems from two primary challenges: a domain discrepancy between clean pre-training data and noisy target data, and a severe scarcity of high-quality, labeled informal datasets.

Standard transfer learning approaches, such as simple fine-tuning or multi-task learning, often struggle to bridge this gap. Training exclusively on small amounts of informal data leads to overfitting, while incorporating too much formal data can prevent the model from fully adapting to the nuances of the informal domain.

The Data Annealing Method

To address these challenges, researchers proposed Data Annealing (DA), a model-agnostic training procedure inspired by simulated annealing in physics. Rather than training on formal and informal data simultaneously or in a fixed sequence, the DA algorithm gradually shifts the ratio of the training data over time.

The process occurs in three distinct phases:

  1. Phase 1 (Initialization): The training set consists primarily of formal "source" data. This provides the model with a stable, clean parameter initialization.
  2. Phase 2 (Transition): The proportion of formal data is decreased exponentially, while the proportion of informal "target" data increases. This allows the model to explore the parameter space and begin adapting to the new domain.
  3. Phase 3 (Refinement): By the end of training, the model focuses almost entirely on the informal target data to master its specific nuances.

Experimental Setup and Datasets

The study evaluated Data Annealing across three core Natural Language Processing (NLP) tasks: Named Entity Recognition (NER), Part-of-Speech (POS) tagging, and text chunking. The researchers utilized several benchmarks to represent the formal/informal divide:

  • Formal (Source) Datasets: OntoNotes-nw (NER), Penn Treebank (POS), and CoNLL 2000 (Chunking).
  • Informal (Target) Datasets: The Ritter11 suite, which consists of Twitter-based datasets notoriously difficult due to their noisy nature.

Performance and State-of-the-Art Results

The "DA BERT-Large" model (Data Annealing applied to BERT-Large) established new state-of-the-art (SOTA) records across all three informal language tasks. As shown in the results table below, Data Annealing consistently outperformed standard fine-tuning ("Vanilla"), Multi-Task Learning (MULT), and simple Parameter Initialization (INIT).

Table 2: Performance comparison across different models and tasks, showing DA BERT-Large achieving new state-of-the-art results.

Specifically, the DA BERT-Large model achieved an F1 score of 69.69 on NER (a +3.16 improvement over previous SOTA), a 92.54 accuracy on POS tagging (+1.37), and an 87.53 F1 score on Chunking (+3.03).

Key Insights

Robustness to Data Scarcity

A critical finding of the research is that Data Annealing remains effective even when labeled informal data is extremely limited. The method maintains a performance lead over vanilla fine-tuning even when using as little as 10% of the available informal training data.

Figure 1: F-1 score vs. dataset proportion, demonstrating that Data Annealing (DA) consistently outperforms Vanilla and INIT methods even with limited data.

Boosting Recall in NER

Data annealing significantly improved recall in Named Entity Recognition. Informal text often contains entities that do not follow standard capitalization or naming conventions. The annealing process provides the model with the "freedom to explore" the parameter space during the transition phase, allowing it to identify these non-traditional entities more effectively than rigid fine-tuning methods.

Conclusion and Future Work

Data Annealing offers a simple yet powerful modification to the data feeding pipeline that significantly enhances a model's ability to handle noisy, informal text. While it sets new benchmarks, challenges remain in handling extremely rare entity types and extreme linguistic noise. Future research will explore assigning higher penalties to infrequent entity types during the refinement phase to further improve performance on the "long tail" of social media language.

Summary updated

Abstract

There is a huge performance gap between formal and informal language understanding tasks. The recent pre-trained models that improved the performance of formal language understanding tasks did not achieve a comparable result on informal language. We pro-pose a data annealing transfer learning procedure to bridge the performance gap on informal natural language understanding tasks. It successfully utilizes a pre-trained model such as BERT in informal language. In our data annealing procedure, the training set contains mainly formal text data at first; then, the proportion of the informal text data is gradually increased during the training process. Our data annealing procedure is model-independent and can be applied to various tasks. We validate its effectiveness in exhaustive experiments. When BERT is implemented with our learning procedure, it outperforms all the state-of-the-art models on the three common informal language tasks.
BibTeX
@inproceedings{gu2020data,
  author = {Jing Gu and Zhou Yu},
  title = {{Data Annealing for Informal Language Understanding Tasks}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2020},
  url = {https://openreview.net/forum?id=7j5qDTJLR3A},
}
Figure