TL;DR
This paper introduces a simple yet effective method to boost BERT's text classification performance by incorporating label embeddings directly into the input. By treating class labels as text and feeding them alongside the document into BERT, the model learns a richer representation that significantly improves accuracy, especially on smaller datasets, without increasing computational cost.
What Problem Does This Paper Address?
Text classification, a fundamental NLP task, relies heavily on effective text representations. While pre-trained models like BERT have revolutionized this field by providing powerful contextualized embeddings, there's still room for improvement, particularly in how class labels are handled. Traditional approaches often treat labels as simple one-hot vectors or rely on separate mechanisms to incorporate them. This can lead to suboptimal performance, especially when dealing with datasets that have limited training examples or when labels themselves carry rich semantic meaning. The paper aims to bridge this gap by proposing a method that seamlessly integrates label semantics into BERT's existing architecture, allowing for a more holistic understanding of both the document and its potential categories.
Methods
The core of this paper's contribution lies in its novel input representation for BERT. Instead of just feeding the document text, the authors propose concatenating the text of the class labels with the document, separated by a [SEP] token. This creates a "sentence pair" input, similar to BERT's original pre-training task.
Here's a breakdown of their approach:
- Label Embedding: The text of each class label (e.g., "world," "sports") is tokenized, and its embedding is calculated by averaging the embeddings of its subwords. This creates a single embedding representation for each label.
- Concatenated Input: The document text and the label text are combined into a single input sequence for BERT. For example, for a document to be classified into one of three classes (L1, L2, L3), the input might look like
[CLS] L1 [SEP] Document [SEP]. - Segment Embeddings: To distinguish between the label text and the document text, different segment embeddings are used, leveraging BERT's inherent ability to handle sentence pairs.
- Two Variants:
- w/ [SEP]: This variant uses the
[SEP]token to explicitly separate the label text from the document text.
- w/ [SEP]: This variant uses the
- w/o [SEP]: This variant concatenates the label text directly before the document text without an explicit
[SEP]token, aiming to simplify the input and potentially avoid pre-training/fine-tuning discrepancies. - TF-IDF Enhancement: To further enrich the label representation, the authors experiment with adding supplemental label texts. They select top-ranked subwords based on their average TF-IDF scores within each class and append them to the original label text. This aims to capture more nuanced semantic aspects of the labels.
Crucially, these methods do not alter BERT's underlying encoder architecture, making them computationally efficient and easy to implement.
Datasets
The authors evaluated their method on six widely-used text classification benchmark datasets:
- AGNews: A news article dataset with titles and descriptions, categorized into four topics (World, Sports, Business, Science & Technology).
- DBPedia: An ontology classification dataset with 14 classes.
- Yahoo! Answers Topic: A dataset containing question titles, content, and best answers, with ten categories.
- IMDb: A binary sentiment classification dataset of movie reviews.
- Yelp Review Full: A dataset extracted from Yelp reviews, with five sentiment classes.
- Yelp Review Polarity: A binary sentiment classification dataset from Yelp reviews.
They also mention creating development sets by randomly splitting the original training data to ensure class distribution consistency with the test sets, as some original benchmarks lacked development sets.
Key Insights and Ideas
- Label Semantics Matter: The paper strongly suggests that explicitly encoding the semantic meaning of class labels into the model's input can significantly improve classification performance. This goes beyond treating labels as mere identifiers.
- Simplicity is Effective: The proposed method is remarkably simple, requiring only a change in input formatting and leveraging BERT's existing self-attention mechanisms. This "plug-and-play" nature makes it highly practical.
- "w/o [SEP]" Outperforms "w/ [SEP]": Interestingly, the variant without the explicit
[SEP]token between labels and documents generally performed better. The authors hypothesize this is due to avoiding potential skewness introduced by the[SEP]token in a non-natural language sequence context, leading to a more stable fine-tuning process. - TF-IDF Augmentation is Beneficial (Mostly): Adding supplemental label texts derived from TF-IDF scores generally boosted performance, particularly on datasets like Yahoo! Answers. This highlights the value of richer label representations.
- Dataset Granularity is Key: The effectiveness of TF-IDF augmentation varied. It was highly beneficial for topic classification (AGNews, Yahoo!) but less so for fine-grained sentiment analysis (Yelp F.), suggesting that TF-IDF might capture topic importance better than nuanced emotional polarity.
- Visualizing Embeddings: t-SNE visualizations of the learned representations showed that label embeddings became more separated in the vector space when fused with BERT, indicating a clearer distinction between classes.
Key Results
The proposed methods consistently improved accuracy over the baseline BERT model on most datasets.
- General Improvement: The "Ours w/ [SEP]" and "Ours w/o [SEP]" methods showed improvements across several datasets.
- Yahoo! Answers Boost: The TF-IDF enhanced versions, especially "Ours w/o [SEP] + tf-idf," significantly boosted performance on the Yahoo! Answers dataset, exceeding baselines.
- Yelp F. Nuance: While improvements were seen on IMDb and Yelp Polarity, Yelp F. showed a performance drop with TF-IDF augmentation. The authors attribute this to the nature of TF-IDF scores being less effective for capturing the subtle emotional nuances in Yelp F. compared to topic distinctions.
- Statistical Significance: Several improvements were statistically significant (p<0.05) when tested using paired bootstrap resampling.
Limitations and Future Work
- Dataset Size Dependency: The paper notes that for very large datasets like Yahoo!, the original label texts might not be discriminative enough, potentially corrupting the input and leading to accuracy degradation. This suggests that the effectiveness of label embedding might be more pronounced on smaller to medium-sized datasets.
- TF-IDF Limitations: The authors acknowledge that TF-IDF might not be the optimal method for constructing enhanced label representations, especially for tasks requiring fine-grained semantic understanding like sentiment analysis.
- Future Directions: The authors express interest in exploring more sophisticated methods for constructing enhanced label representations and investigating different ways to uncover the full potential of pre-trained attentional models like BERT. They also suggest that the "w/o [SEP]" approach, which avoids potential pre-training/fine-tuning discrepancies, could lead to more stabilized improvements.
Summary updated