Label-guided Learning for Text Classification

Xien Liu, Song Wang, Xiao Zhang, Xinxin You, Ji Wu, Dejing DouarXiv 2020 · arXiv

Introduction

In Natural Language Processing (NLP), text classification is a fundamental task. The standard paradigm typically involves encoding text into a vector using neural architectures like CNNs or LSTMs, followed by a classification layer. However, this approach often learns what the text says before considering what the text is about. The paper "Label-guided Learning for Text Classification" proposes a shift toward LguidedLearn, a framework that uses label information to guide the text encoding process itself.

Problem Statement

Most existing models focus on local contextual information—how words relate to their immediate neighbors. While effective, these models often ignore global clues provided by the labels. In a standard setup, a model encodes a sentence into a "universal" embedding regardless of the classification objective. LguidedLearn aims to provide neural networks with a "label-aware" focus, similar to how a human reader tunes into specific keywords when looking for a particular topic.

Methodology

The authors propose a four-layer architecture that can be integrated with various text encoders:

  1. Pre-trained Layer: Utilizes standard word embeddings (e.g., GloVe) for initial word vectors.
  2. Contextual Encoding Layer: Employs a BiLSTM or BERT to capture sequential word relationships.
  3. Label-guided Encoding Layer: The core innovation. Instead of representing each label as a single point, labels are represented as an embedding matrix. This allows a label to have multiple "facets" (e.g., "Sports" capturing teams, equipment, and scores). An attention mechanism determines how well each word matches these label facets.
  4. Classifying Layer: A Multi-Layer Perceptron (MLP) that processes label-wise embeddings for final prediction.
The LguidedLearn architecture featuring the label-guided encoding layer.

Datasets

The framework was evaluated on two major collections:

  • TCD-1: Standard benchmarks including DBPedia (ontology), Yelp-F/B (sentiment), YahooQA (topic), and AGNews (news).
  • TCD-2: Specialized datasets including 20NG (newsgroups), R8/R52 (Reuters), Ohsumed (medical), and MR (movie reviews).

Key Insights

  • Labels as Features: By treating labels as entities with their own embeddings, the model performs "label-word attentive encoding."
  • The Power of the Matrix: Using multiple vectors per label (a matrix) captures linguistic diversity within a single category more effectively than a single vector.
  • Plug-and-Play Compatibility: The label-guided layer is architecture-agnostic, providing performance boosts for both BiLSTMs and BERT.
  • Discriminative Robustness: The resulting embeddings are more discriminative, increasing the mathematical distance between different classes in the vector space.

Key Results

LguidedLearn achieved state-of-the-art or near-state-of-the-art results across various benchmarks:

  • Superiority over Baselines: It consistently outperformed standard models like fastText, CNNs, and LSTMs.
  • Beating Graph Networks: On datasets like R8 and R52, it outperformed Text GCN, a more computationally expensive graph-based model.
  • Significant Gains on Complex Tasks: On Ohsumed and MR datasets, the label-guided layer provided over a 5% improvement compared to standard BiLSTMs.
  • Optimal Complexity: Ablation studies indicated that using 5 embeddings per label provides an optimal balance between complexity and accuracy.
Performance comparison of LguidedLearn against various baselines on TCD-1 datasets.

Limitations and Future Work

Despite its strengths, LguidedLearn faces challenges with extremely long documents (e.g., 20NG), where graph-based approaches like Text GCN may still hold an advantage. Additionally, the number of embeddings per label is a hyperparameter that requires tuning. Future research may involve incorporating explicit label descriptions to further refine the label matrices.

Summary updated

Abstract

Text classification is one of the most important and fundamental tasks in natural language processing. Performance of this task mainly dependents on text representation learning. Currently, most existing learning frameworks mainly focus on encoding local contextual information between words. These methods always neglect to exploit global clues, such as label information, for encoding text information. In this study, we propose a label-guided learning framework LguidedLearn for text representation and classification. Our method is novel but simple that we only insert a label-guided encoding layer into the commonly used text representation learning schemas. That label-guided layer performs label-based attentive encoding to map the universal text embedding (encoded by a contextual information learner) into different label spaces, resulting in label-wise embeddings. In our proposed framework, the label-guided layer can be easily and directly applied with a contextual encoding method to perform jointly learning. Text information is encoded based on both the local contextual information and the global label clues. Therefore, the obtained text embeddings are more robust and discriminative for text classification. Extensive experiments are conducted on benchmark datasets to illustrate the effectiveness of our proposed method.
BibTeX
@misc{liu2020label,
  author = {Xien Liu and Song Wang and Xiao Zhang and Xinxin You and Ji Wu and Dejing Dou},
  title = {{Label-guided Learning for Text Classification}},
  year = {2020},
  eprint = {2002.10772},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2002.10772},
}
Figure