Label Confusion Learning to Enhance Text Classification Models

Biyang Guo, Songqiao Han, Xiao Han, Hailiang Huang, Ting LuAAAI 2021 · DOI

Introduction

Traditional text classification models typically employ one-hot encoding for labels, assuming that categories are mutually exclusive and independent. However, this rigid approach fails to capture the inherent semantic overlaps between categories—for instance, a news article about "Smartphones" is naturally related to both "Electronics" and "Technology." This "Label Confusion Problem" (LCP) can lead to over-confident models that struggle with ambiguous samples or noisy data. To address this, researchers from the Shanghai University of Finance and Economics proposed the Label Confusion Model (LCM) at AAAI-21.

The Label Confusion Model (LCM) Architecture

The LCM is designed as a plug-and-play enhancement for existing text classification architectures, such as LSTMs, CNNs, or BERT. It operates by learning a "Simulated Label Distribution" (SLD) that reflects the semantic similarities between labels for each specific input instance.

The architecture of the Label Confusion Model (LCM) alongside a basic predictor.

As shown in Figure 1, the system consists of two parallel tracks:

  1. Basic Predictor: A standard deep representation network (e.g., BERT) followed by a softmax classifier that produces a predicted label distribution (PLD).
  2. LCM Component: This branch includes a Label Encoder that generates vector representations for each category. A Similarity Layer then computes the relationship between the input instance's representation and these label embeddings to produce a "label confusion vector."

Simulated Label Distribution (SLD)

The core innovation of LCM is the generation of the SLD. Instead of training against a hard one-hot vector, the model is trained to minimize the KL-Divergence between its prediction (PLD) and the SLD. The SLD is a weighted blend of the ground-truth one-hot vector and the learned confusion vector. A hyperparameter, $\alpha$, controls the degree of "softness": a higher $\alpha$ keeps the distribution closer to the one-hot label, while a lower $\alpha$ allows the model to incorporate more information about related categories. Crucially, this distribution is instance-dependent, meaning it changes based on the specific content of the text.

Experimental Evaluation

The researchers evaluated LCM across five major benchmarks, including 20 Newsgroups (20NG), AG’s News, and DBPedia, as well as Chinese datasets like THUCNews.

The results demonstrated consistent performance gains across all models. For example, adding LCM to a standard LSTM on the 20NG dataset resulted in a 4.20% accuracy increase. Furthermore, LCM showed superior robustness to label noise. In experiments where 30% of the training labels were intentionally corrupted, the LCM-enhanced BERT significantly outperformed the baseline, suggesting that the model can leverage semantic similarities to "see through" labeling errors.

Visualizing Learned Label Relationships

One of the most compelling aspects of LCM is its ability to learn meaningful label relationships from scratch, even without pre-defined hierarchies.

Visualization of learned label relationships through a cosine similarity matrix and t-SNE projection.

Figure 6 illustrates this through a cosine similarity matrix and a t-SNE visualization of the label representations. The model successfully clusters semantically related categories—such as grouping various computer-related subcategories (e.g., comp.sys.ibm.pc.hardware and comp.sys.mac.hardware) or sports categories. This confirms that the LCM effectively captures the semantic overlaps inherent in human language.

Conclusion and Future Work

The Label Confusion Model offers a robust alternative to rigid one-hot labeling and uniform label smoothing. By making the label distribution instance-dependent and semantically aware, LCM improves both accuracy and generalization. Notably, the LCM component is only required during training; it can be discarded during inference, ensuring no additional computational overhead in production environments. Future research directions include extending this framework to multi-label classification and exploring more sophisticated label encoding techniques.

Reference: Guo, B., Han, S., Han, X., Huang, H., & Lu, T. (2021). Label Confusion Learning to Enhance Text Classification Models. Proceedings of the AAAI Conference on Artificial Intelligence.

Summary updated

BibTeX
@inproceedings{guo2021label,
  author = {Biyang Guo and Songqiao Han and Xiao Han and Hailiang Huang and Ting Lu},
  title = {{Label Confusion Learning to Enhance Text Classification Models}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2021},
  doi = {10.1609/aaai.v35i14.17529},
}
Figure