Out-of-Manifold Regularization in Contextual Embedding Space for Text Classification

Seonghyeon Lee, Dongha Lee, Hwanjo Yu

Introduction

Large-scale pre-trained language models like BERT have revolutionized natural language processing. However, these models are often significantly over-parameterized. While they operate in high-dimensional vector spaces (e.g., 768 dimensions for BERT-base), research indicates that actual word embeddings occupy only a small, low-dimensional "manifold" within that space. For instance, it has been observed that approximately 33% of BERT's dimensions can account for 80% of its variance. This leaves a vast "out-of-manifold" (OoM) space that remains unexplored during standard training, leading to potential overfitting and inefficient use of the model's capacity, particularly in low-resource settings.

The OoMMix Framework

To address this inefficiency, researchers from POSTECH introduced OoMMix at ACL-IJCNLP 2021. OoMMix leverages a generator-discriminator framework to synthesize artificial data points in the empty OoM regions of the embedding space, forcing the model to learn more robust decision boundaries.

The OoMMix architecture integrates an embedding generator and a manifold discriminator with the BERT backbone to explore out-of-manifold regions.

The framework consists of three primary components:

  1. Embedding Generator ($G$): This module performs linear interpolation between two real hidden-layer embeddings. Unlike traditional "Mixup" methods that use random mixing coefficients, $G$ is trained to find coefficients that produce synthetic points specifically located outside the natural data manifold.
  2. Manifold Discriminator ($D$): A binary classifier designed to distinguish between "real" embeddings (derived from actual text) and "synthetic" embeddings produced by the generator.
  3. Joint Optimization: The system optimizes a composite loss function including standard classification loss, a generator loss (encouraging the creation of helpful but OoM points), and a discriminator loss (refining the manifold boundary detection).

Experimental Evaluation

The efficacy of OoMMix was evaluated across four standard text classification benchmarks: AG News (topic), Amazon Review (sentiment), Yahoo Answer (category), and DBpedia (entity). Experiments were conducted across varying training set sizes, ranging from 500 to 35,000 samples, to assess performance in both low- and high-resource scenarios.

Results and Analysis

OoMMix consistently achieved state-of-the-art performance, outperforming existing Mixup variants such as TMix and NonlinearMix. A key finding is the method's effectiveness in low-data regimes; for example, on AG News with 2,500 samples, OoMMix improved baseline accuracy from 89.92% to 90.25%.

Comparison of accuracy across different data augmentation methods using 10K training samples, demonstrating the consistent performance boost provided by OoMMix.

Furthermore, OoMMix proved to be highly complementary to traditional "in-manifold" data augmentation techniques like back-translation (BT) or word swapping (EDA). While these methods modify the input text, OoMMix regularizes the internal embedding space. Combining these approaches yielded superior results, suggesting that targeting both the input and the latent manifold provides a more comprehensive regularization effect.

Conclusion

OoMMix demonstrates that the "empty" space in high-dimensional embedding models is not merely a byproduct of over-parameterization but a valuable resource for regularization. By actively exploring the out-of-manifold regions, OoMMix enables models like BERT to achieve greater robustness and better performance, especially when training data is scarce.

Summary updated

BibTeX
@inproceedings{lee2021out,
  author = {Seonghyeon Lee and Dongha Lee and Hwanjo Yu},
  title = {{Out-of-Manifold Regularization in Contextual Embedding Space for Text Classification}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
  doi = {10.18653/v1/2021.acl-long.49},
}
Figure