MixText: Linguistically-Informed Interpolation of Hidden Space for Semi-Supervised Text Classification

Jiaao Chen, Zichao Yang, Diyi Yang

Abstract

This paper presents MixText, a semi-supervised learning method for text classification, which uses our newly designed data augmentation method called TMix. TMix creates a large amount of augmented training samples by interpolating text in hidden space. Moreover, we leverage recent advances in data augmentation to guess low-entropy labels for unlabeled data, hence making them as easy to use as labeled data. By mixing labeled, unlabeled and augmented data, MixText significantly outperformed current pre-trained and fined-tuned models and other state-of-the-art semi-supervised learning methods on several text classification benchmarks. The improvement is especially prominent when supervision is extremely limited. We have publicly released our code at https://github.com/GT-SALT/MixText.

Introduction

Deep learning models are notoriously data-hungry. While massive amounts of unlabeled text are readily available, labeling that data remains expensive and time-consuming. In "low-resource" settings—where only a handful of labeled examples are available—standard supervised models like BERT tend to overfit, memorizing small training sets rather than learning generalizable patterns.

Previous semi-supervised learning (SSL) methods often treated labeled and unlabeled data separately, lacking a clear mechanism to "transit" supervision between them. MixText addresses this gap by creating a continuous space between data points, forcing the model to learn the relationships between samples rather than just the samples themselves.

Methodology

The core of the MixText framework is a two-pronged approach combining novel interpolation with robust label guessing.

TMix: Hidden Space Interpolation

Inspired by "Mixup" in computer vision, TMix applies interpolation to text. Since discrete words cannot be easily mixed, TMix passes two sentences through a BERT encoder and interpolates their vector representations at an intermediate layer (specifically layers 7, 9, or 12). If Sentence A (Label: Sports) and Sentence B (Label: Politics) are mixed with a 50/50 weight, the model is trained to predict a 50/50 label distribution for the resulting mixed vector.

The TMix mechanism interpolates hidden representations at an intermediate layer to create new training signals.

The MixText SSL Framework

Beyond TMix, the framework incorporates several SSL strategies:

  • Data Augmentation: Unlabeled sentences are augmented using back-translation (e.g., English to German and back) to create semantic variations.
  • Label Guessing & Sharpening: The model predicts labels for these augmented versions, averages them, and applies a "sharpening" function to produce a confident pseudo-label.
  • Joint Training: The model performs TMix across the union of labeled and unlabeled data, learning a smoother decision boundary by understanding how sentences gradually transform between classes in vector space.

Experimental Results

The authors evaluated MixText on four standard benchmarks: AG News, DBpedia, Yahoo! Answers, and IMDB. They specifically tested scenarios with 10, 200, and 2,500 labeled examples per class.

MixText consistently outperforms baselines across all datasets, with the most significant gains in extreme low-data regimes.

MixText consistently outperformed baselines like VAMPIRE, standard BERT, and UDA (Unsupervised Data Augmentation). The results were most striking in cases of extreme scarcity; on AG News with only 10 labels per class, MixText achieved 88.4% accuracy, compared to standard BERT's 69.5%.

Key Insights

  • Layer Selection: Mixing at the input level (word embeddings) is less effective than mixing at deeper layers (7, 9, and 12), which capture a mix of complex syntax and semantics.
  • Regularization: Ablation studies showed that TMix acts as a powerful regularizer. While standard BERT models show volatile loss curves on small datasets, MixText maintains a much more stable and lower loss on development sets.

Conclusion

MixText provides a robust solution for text classification when labeled data is scarce. While it relies on high-quality back-translation and incurs higher computational costs than simple fine-tuning, its ability to leverage hidden-space interpolation significantly advances the state-of-the-art in semi-supervised learning. Future work may explore its application to sequential labeling tasks like Named Entity Recognition (NER).

Summary updated

Abstract

This paper presents MixText, a semi-supervised learning method for text classification, which uses our newly designed data augmentation method called TMix. TMix creates a large amount of augmented training samples by interpolating text in hidden space. Moreover, we leverage recent advances in data augmentation to guess low-entropy labels for unlabeled data, hence making them as easy to use as labeled data.By mixing labeled, unlabeled and augmented data, MixText significantly outperformed current pre-trained and fined-tuned models and other state-of-the-art semi-supervised learning methods on several text classification benchmarks. The improvement is especially prominent when supervision is extremely limited. We have publicly released our code at https://github.com/GT-SALT/MixText.
BibTeX
@inproceedings{chen2020mixtext,
  author = {Jiaao Chen and Zichao Yang and Diyi Yang},
  title = {{MixText: Linguistically-Informed Interpolation of Hidden Space for Semi-Supervised Text Classification}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2020},
  url = {https://openreview.net/forum?id=5MTi6yj4-No},
}

Loading related content...

Figure