Data Annealing for Informal Language Understanding Tasks

Jing Gu, Zhou Yu

Abstract

There is a huge performance gap between formal and informal language understanding tasks. The recent pre-trained models that improved the performance of formal language understanding tasks did not achieve a comparable result on informal language. We propose a data annealing transfer learning procedure to bridge the performance gap on informal natural language understanding tasks. It successfully utilizes a pre-trained model such as BERT in informal language. In our data annealing procedure, the training set contains mainly formal text data at first; then, the proportion of the informal text data is gradually increased during the training process. Our data annealing procedure is model-independent and can be applied to various tasks. We validate its effectiveness in exhaustive experiments. When BERT is implemented with our learning procedure, it outperforms all the state-of-the-art models on the three common informal language tasks.

Introduction

Large-scale pre-trained models like BERT have revolutionized Natural Language Processing (NLP), but they suffer from a significant "formal bias." Because these models are primarily trained on structured text like Wikipedia and books, their performance drops sharply when applied to informal language—such as the slang, typos, and idiosyncratic grammar found on social media platforms like Twitter.

This performance gap stems from two main challenges:

  1. Domain Discrepancy: The massive linguistic distance between "clean" news text and "noisy" social media content.
  2. Data Scarcity: A significant shortage of high-quality labeled informal data compared to the abundance of formal datasets.

Simply fine-tuning a formal model on informal data often leads to sub-optimal results, as the model struggles to navigate the noisy landscape of informal text from a rigid formal starting point.

Methodology: Data Annealing

To address these issues, we introduce Data Annealing (DA), a model-agnostic training procedure inspired by simulated annealing in physics and learning rate decay in optimization. The core concept is to control the "temperature" or composition of the training data over time.

The DA procedure follows three stages:

  • Initialization: The training set consists mostly of source data (formal, clean text). This provides the model with a stable parameter initialization and a "clean" environment to explore update directions.
  • Transition: As training progresses, the proportion of source data is reduced using an exponential decay function, while the proportion of target data (informal, noisy text) is increased.
  • Finalization: In the final stages, the model focuses almost entirely on the target informal data to exploit the specific nuances of that domain.

This approach was tested using both Bi-LSTM and BERT architectures, with a Conditional Random Field (CRF) layer added for sequence labeling tasks.

Experimental Setup

The method was evaluated across three common NLP tasks, pairing a formal "source" dataset with an informal "target" dataset:

  1. Named Entity Recognition (NER): Source: OntoNotes-nw (News); Target: RitterNER (Twitter).
  2. Part-of-Speech (POS) Tagging: Source: Penn Treebank (PTB); Target: RitterPOS (Twitter).
  3. Chunking: Source: CoNLL 2000; Target: RitterCHUNK (Twitter).

Results

The "DA BERT-Large" model outperformed all previous state-of-the-art (SOTA) models across all three tasks. As shown in the table below, the improvements were substantial, particularly in F1 scores for NER and Chunking.

Performance comparison of various models across NER, POS, and Chunking tasks.

Specifically, DA BERT-Large achieved:

  • NER: 69.69 $F_1$ score (+3.16 points over SOTA).
  • POS Tagging: 92.54% accuracy (+1.37% over SOTA).
  • Chunking: 87.53 $F_1$ score (+3.03 points over SOTA).

Notably, the method proved highly robust in low-resource scenarios. Even when using only 10% of the available informal training data, the Data Annealing approach significantly outperformed vanilla BERT fine-tuning, demonstrating its efficiency in data-scarce environments.

F-1 score performance of BERT-Large variants across different proportions of the informal dataset.

Discussion and Key Insights

Data Annealing works by providing guided exploration. Much like a decaying learning rate allows a model to explore the loss landscape broadly before settling, DA uses clean data to guide the model toward a good neighborhood in the parameter space before introducing the noise of informal text.

Furthermore, the source data acts as a regularizer, preventing the model from overfitting to small, noisy target datasets. By gradually shifting the data distribution rather than switching abruptly, the model avoids "catastrophic forgetting," retaining general linguistic knowledge while slowly adapting to informal quirks. A key finding in NER was that DA significantly boosted recall, helping the model identify entities it would have otherwise ignored.

Conclusion

Data Annealing is a simple yet powerful transfer learning technique that bridges the performance gap between formal and informal language tasks. While the model shows significant gains, some informal sentences remain extremely challenging due to heavy slang or typos. Future work may involve incorporating dedicated denoising steps or pre-training models directly on noisy text to further push the boundaries of informal language understanding.

Summary updated

Abstract

There is a huge performance gap between formal and informal language understanding tasks. The recent pre-trained models that improved the performance of formal language understanding tasks did not achieve a comparable result on informal language. We pro-pose a data annealing transfer learning procedure to bridge the performance gap on informal natural language understanding tasks. It successfully utilizes a pre-trained model such as BERT in informal language. In our data annealing procedure, the training set contains mainly formal text data at first; then, the proportion of the informal text data is gradually increased during the training process. Our data annealing procedure is model-independent and can be applied to various tasks. We validate its effectiveness in exhaustive experiments. When BERT is implemented with our learning procedure, it outperforms all the state-of-the-art models on the three common informal language tasks.
BibTeX
@inproceedings{gu2020data,
  author = {Jing Gu and Zhou Yu},
  title = {{Data Annealing for Informal Language Understanding Tasks}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2020},
  url = {https://openreview.net/forum?id=7j5qDTJLR3A},
}
Figure