Introduction
Large-scale pre-trained language models like BERT have revolutionized natural language processing. However, these models are often significantly over-parameterized. While they operate in high-dimensional vector spaces (e.g., 768 dimensions for BERT-base), research indicates that actual word embeddings occupy only a small, low-dimensional "manifold" within that space. For instance, it has been observed that approximately 33% of BERT's dimensions can account for 80% of its variance. This leaves a vast "out-of-manifold" (OoM) space that remains unexplored during standard training, leading to potential overfitting and inefficient use of the model's capacity, particularly in low-resource settings.
The OoMMix Framework
To address this inefficiency, researchers from POSTECH introduced OoMMix at ACL-IJCNLP 2021. OoMMix leverages a generator-discriminator framework to synthesize artificial data points in the empty OoM regions of the embedding space, forcing the model to learn more robust decision boundaries.
The framework consists of three primary components:
- Embedding Generator ($G$): This module performs linear interpolation between two real hidden-layer embeddings. Unlike traditional "Mixup" methods that use random mixing coefficients, $G$ is trained to find coefficients that produce synthetic points specifically located outside the natural data manifold.
- Manifold Discriminator ($D$): A binary classifier designed to distinguish between "real" embeddings (derived from actual text) and "synthetic" embeddings produced by the generator.
- Joint Optimization: The system optimizes a composite loss function including standard classification loss, a generator loss (encouraging the creation of helpful but OoM points), and a discriminator loss (refining the manifold boundary detection).
Experimental Evaluation
The efficacy of OoMMix was evaluated across four standard text classification benchmarks: AG News (topic), Amazon Review (sentiment), Yahoo Answer (category), and DBpedia (entity). Experiments were conducted across varying training set sizes, ranging from 500 to 35,000 samples, to assess performance in both low- and high-resource scenarios.
Results and Analysis
OoMMix consistently achieved state-of-the-art performance, outperforming existing Mixup variants such as TMix and NonlinearMix. A key finding is the method's effectiveness in low-data regimes; for example, on AG News with 2,500 samples, OoMMix improved baseline accuracy from 89.92% to 90.25%.
Furthermore, OoMMix proved to be highly complementary to traditional "in-manifold" data augmentation techniques like back-translation (BT) or word swapping (EDA). While these methods modify the input text, OoMMix regularizes the internal embedding space. Combining these approaches yielded superior results, suggesting that targeting both the input and the latent manifold provides a more comprehensive regularization effect.
Conclusion
OoMMix demonstrates that the "empty" space in high-dimensional embedding models is not merely a byproduct of over-parameterization but a valuable resource for regularization. By actively exploring the out-of-manifold regions, OoMMix enables models like BERT to achieve greater robustness and better performance, especially when training data is scarce.
Summary updated