TIER-A: Denoising Learning Framework for Information Extraction

Yongkang Li, Ming ZhangarXiv 2022 · OpenReview

Abstract

With the development of deep neural language models, great progress has been made in information extraction recently. However, deep learning models often overfit on noisy data points, leading to poor performance. In this work, we examine the role of information entropy in the overfitting process and draw a key insight that overfitting is a process of overconfidence and entropy decreasing. Motivated by such properties, we propose a simple yet effective co-regularization joint-training framework TIER-A, Aggregation Joint-training Framework with Temperature Calibration and Information Entropy Regularization. Our framework consists of several neural models with identical structures. These models are jointly trained and we avoid overfitting by introducing temperature and information entropy regularization. Extensive experiments on two widely-used but noisy datasets, TACRED and CoNLL03, demonstrate the correctness of our assumption and the effectiveness of our framework.

Introduction

Information Extraction (IE)—tasks like identifying entities (NER) or the relationships between them (RE)—is the backbone of many NLP pipelines. However, large-scale datasets are rarely perfect; they are often riddled with "noisy" labels (incorrectly annotated data). Modern deep learning models are exceptionally proficient at fitting data, which unfortunately includes these errors. This leads to overfitting, where the model becomes "overconfident" in its predictions on the training set but fails to generalize to new, clean data.

TIER-A is a new joint-training framework designed to prevent IE models from "memorizing" noisy or incorrect labels. By treating overfitting as a process of increasing overconfidence and decreasing entropy, the authors use temperature calibration and entropy regularization to keep models robust, achieving significant performance gains on real-world, messy datasets.

The TIER-A Framework

The authors introduce TIER-A (Aggregation Joint-training Framework with Temperature Calibration and Information Entropy Regularization). The framework architecture is designed to maintain a "healthy doubt" during training to avoid premature convergence on noisy labels.

The TIER-A joint-training framework architecture.

The framework operates through several key mechanisms:

  1. Multi-Model Ensemble: It trains $n$ copies (usually just two) of the same base model (like BERT or LUKE) with different initializations.
  2. Two-Phase Training: A warm-up phase allows models to capture main patterns independently, followed by a joint-training phase where models are encouraged to agree via a shared "soft label."
  3. Temperature Calibration ($T$): By scaling the model's output (logits) by a temperature factor before the softmax layer, the framework controls the "sharpness" of the probability distribution, preventing overconfidence.
  4. Negative Information Entropy (NIE) Regularization: This penalty explicitly discourages low entropy, forcing the model to maintain uncertainty.
  5. Confidence Soft Vote: During inference, a weighted voting system based on normalized logits allows robust model copies to outvote those that may have overfitted to specific noise.

Datasets and Evaluation

The framework was evaluated on two major benchmarks known to contain significant noise:

  • TACRED: A large-scale Relation Extraction dataset with approximately 6.62% noise.
  • CoNLL03: A classic Named Entity Recognition dataset with approximately 5.38% noise.

To ensure the models were learning underlying truths rather than the noisy training distribution, the authors evaluated performance against "cleaned" (relabeled) versions of these test sets.

Key Insights and Results

The core insight of this work is that overfitting is essentially a thermodynamic-like process where entropy decreases as softmax outputs creep toward 1.0. By using low temperature and entropy regularization, TIER-A creates a "stalemate" between the true class and the noisy label, giving the model more time to learn difficult, legitimate "long-tail" classes.

Performance comparison of TIER-A against various baselines on the TACRED dataset.

Experimental results demonstrate that TIER-A consistently outperforms baselines like NLLIE and CrossWeight. On the TACRED relabeled test set, it improved F1 scores by 3.1% to 3.8% across various base models (ALBERT, BERT, LUKE). Furthermore, TIER-A showed remarkable resistance to extreme noise; in stress tests with 50% flipped labels, it maintained an F1 score of ~74.5%, significantly outperforming standard BERT models.

Conclusion

TIER-A offers a practical and efficient solution for denoising in IE. Unlike previous methods requiring dozens of model copies, TIER-A achieves state-of-the-art results with just 2 or 3 copies. While it introduces some computational overhead during training, its ability to recover patterns even under high noise levels makes it a robust choice for real-world applications. Future work may explore dynamic adjustments of temperature and regularization based on batch-specific noise estimates.

Summary updated

Abstract

With the development of deep neural language models, great progress has been made in information extraction recently. However, deep learning models often overfit on noisy data points, leading to poor performance. In this work, we examine the role of information entropy in the overfitting process and draw a key insight that overfitting is a process of overconfidence and entropy decreasing. Motivated by such properties, we propose a simple yet effective co-regularization joint-training framework TIER-A, Aggregation Joint-training Framework with Temperature Calibration and Information Entropy Regularization. Our framework consists of several neural models with identical structures. These models are jointly trained and we avoid overfitting by introducing temperature and information entropy regularization. Extensive experiments on two widely-used but noisy datasets, TACRED and CoNLL03, demonstrate the correctness of our assumption and the effectiveness of our framework.
BibTeX
@misc{li2022tier,
  author = {Yongkang Li and Ming Zhang},
  title = {{TIER-A: Denoising Learning Framework for Information Extraction}},
  year = {2022},
  eprint = {2211.11527},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2211.11527},
}
Figure