MixText: Linguistically-Informed Interpolation of Hidden Space for Semi-Supervised Text Classification

Jiaao Chen, Zichao Yang, Diyi Yang

Abstract

This paper presents MixText, a semi-supervised learning method for text classification, which uses our newly designed data augmentation method called TMix. TMix creates a large amount of augmented training samples by interpolating text in hidden space. Moreover, we leverage recent advances in data augmentation to guess low-entropy labels for unlabeled data, hence making them as easy to use as labeled data. By mixing labeled, unlabeled and augmented data, MixText significantly outperformed current pre-trained and fined-tuned models and other state-of-the-art semi-supervised learning methods on several text classification benchmarks. The improvement is especially prominent when supervision is extremely limited.

Introduction

Deep learning models, particularly Transformer-based architectures like BERT, have revolutionized NLP but remain notoriously "data-hungry." In many practical applications, while raw text is abundant, high-quality human-annotated labels are scarce and expensive to obtain. Existing semi-supervised learning (SSL) methods often struggle to effectively bridge the gap between labeled and unlabeled data, frequently leading to overfitting on small labeled sets.

MixText addresses this by creating a continuous training space where labeled and unlabeled data are blended. This approach allows supervision to "flow" more effectively across the entire data distribution, significantly improving generalization in low-resource settings.

Methodology

TMix: Hidden Space Interpolation

A common data augmentation technique in computer vision is "Mixup," which creates new training samples by linearly interpolating the pixels of two images. Directly applying this to text is impossible because averaging discrete tokens results in nonsensical input.

TMix overcomes this by performing interpolation in the hidden space. Two text sequences are passed through the initial layers of a BERT model. Their vector representations (hidden states) are then averaged at a specific layer $m$, along with their corresponding labels. The remainder of the network processes this hybrid vector to predict the interpolated label.

Architecture of TMix, performing interpolation in the hidden space of neural network layers.

The MixText Framework

To leverage unlabeled data, MixText employs a multi-step pipeline:

  1. Data Augmentation: Unlabeled sentences undergo "back-translation" (e.g., English to German and back) to generate diverse versions of the same content.
  2. Label Guessing: The model generates predictions for these augmented versions. These predictions are averaged and "sharpened" to produce a low-entropy pseudo-label, encouraging the model to be more confident.
  3. Unified Mixing: These pseudo-labeled examples are treated as ground truth. TMix is then used to blend all available data: labeled with labeled, unlabeled with unlabeled, and crucially, labeled with unlabeled.

Experiments and Results

The authors evaluated MixText on four standard benchmarks: AG News (topic classification), DBpedia (ontology classification), Yahoo! Answers (QA topic classification), and IMDB (sentiment analysis).

MixText demonstrates remarkable performance, particularly when labeled data is extremely limited. For instance, on AG News with only 10 labeled examples per class, MixText achieves 88.4% accuracy, a substantial improvement over standard fine-tuned BERT (69.5%).

Performance comparison of MixText against several baselines across four benchmark datasets with varying amounts of labeled data.

As shown in the figures above, MixText consistently outperforms state-of-the-art SSL methods like UDA (Unsupervised Data Augmentation) and VAMPIRE across different datasets and label counts.

Key Insights

  • Optimal Mixing Layers: The choice of layer $m$ for interpolation is critical. Mixing at very early layers (0-2) yields minimal benefit. The most effective results occur at layers 7, 9, and 12, where BERT captures sophisticated syntactic and semantic features.
  • Linearity as Regularization: By enforcing linear behavior between data points, TMix acts as a powerful regularizer. This prevents the model from memorizing small labeled sets and encourages smoother decision boundaries.
  • Bridging the Supervision Gap: Directly mixing labeled and unlabeled data allows the model to learn implicit relationships between them, maximizing the utility of every labeled example.

Conclusion and Future Work

MixText provides a robust framework for semi-supervised text classification, achieving state-of-the-art results with minimal supervision. While highly effective, the reliance on back-translation introduces computational overhead. Future research could explore extending hidden-space interpolation to more complex structured prediction tasks like Named Entity Recognition or text generation.

Code is available at: https://github.com/GT-SALT/MixText

Summary updated

Abstract

This paper presents MixText, a semi-supervised learning method for text classification, which uses our newly designed data augmentation method called TMix. TMix creates a large amount of augmented training samples by interpolating text in hidden space. Moreover, we leverage recent advances in data augmentation to guess low-entropy labels for unlabeled data, hence making them as easy to use as labeled data. By mixing labeled, unlabeled and augmented data, MixText significantly outperformed current pre-trained and fined-tuned models and other state-of-the-art semi-supervised learning methods on several text classification benchmarks. The improvement is especially prominent when supervision is extremely limited. We have publicly released our code at https://github.com/GT-SALT/MixText.
BibTeX
@inproceedings{chen2020mixtext,
  author = {Jiaao Chen and Zichao Yang and Diyi Yang},
  title = {{MixText: Linguistically-Informed Interpolation of Hidden Space for Semi-Supervised Text Classification}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2020},
  url = {https://openreview.net/forum?id=5MTi6yj4-No},
}
Figure