Abstract
Large pre-trained language models like BERT are often over-parameterized, leading to overfitting during fine-tuning as they develop arbitrary decision boundaries in high-dimensional spaces that lack training data. We introduce OoMMix, a novel regularization technique that targets the "out-of-manifold" space—the regions of the embedding space not occupied by natural language data. By employing a generator-discriminator framework, OoMMix synthesizes informative out-of-manifold embeddings to smooth decision boundaries. Our experiments across four major benchmarks demonstrate that OoMMix consistently improves classification accuracy, especially in low-resource scenarios, and is highly complementary to existing data augmentation methods.
Introduction
While models like BERT produce high-dimensional contextual embeddings (e.g., 768 dimensions), research suggests that real-world word embeddings actually cluster on a much lower-dimensional "manifold." During fine-tuning, the vast majority of the high-dimensional space remains unexplored, allowing the model to create complex, non-robust decision boundaries in these "empty" regions. This phenomenon is a primary driver of overfitting when training data is scarce. While standard data augmentation techniques like back-translation help populate the manifold, they do not address the unconstrained space outside of it.
Methodology: The OoMMix Framework
OoMMix addresses this gap by actively searching for and training on points in the out-of-manifold region. The framework consists of two primary components integrated into the transformer architecture:
- Embedding Generator ($G$): Positioned at an early layer (typically layer 3), the generator takes two real contextual embeddings and learns to produce an optimal mixing coefficient ($\lambda$). Unlike standard Mixup, which uses random interpolation, the generator is trained to find synthetic points that are task-relevant yet challenging for the discriminator.
- Manifold Discriminator ($D$): Attached to the final layer, the discriminator acts as a binary classifier, attempting to distinguish between real word embeddings and the synthetic embeddings produced by the generator.
The Training Objective
The system is trained end-to-end using a minimax-style game. The generator aims to "fool" the discriminator by creating embeddings that appear to be in-manifold but are actually located in the previously unconstrained high-dimensional space. Simultaneously, the classification head is trained to correctly categorize these synthetic points using soft, interpolated labels. This process forces the model to learn smoother decision boundaries that extend reliably into the out-of-manifold regions.
Experimental Results
We evaluated OoMMix on four standard text classification benchmarks: AG News, Amazon Review, Yahoo Answer, and DBpedia. To test robustness, we varied the training set sizes from as few as 500 samples to 35,000 samples.
Performance and Scalability
OoMMix consistently outperformed standard BERT fine-tuning and other state-of-the-art mixup techniques like TMix. The performance gains were most pronounced in low-resource settings. For instance, on the AG News dataset with only 2.5k samples, OoMMix improved accuracy to 90.25% from a baseline of 89.92%.
Synergy with Data Augmentation
A key finding is that OoMMix is complementary to traditional data augmentation (DA) methods such as Easy Data Augmentation (EDA) and Back-translation (BT). While DA strengthens the "in-manifold" representations by providing more linguistic variations, OoMMix regularizes the "out-of-manifold" space. Combining both approaches yielded the highest overall performance across all tested benchmarks.
Conclusion
OoMMix provides a robust solution to the over-parameterization of transformer models by regularizing the high-dimensional space outside the natural data manifold. By replacing random sampling with an adversarial search for informative interpolation points, the method effectively smooths decision boundaries and reduces overfitting. Future work may explore the application of out-of-manifold regularization to more complex sequence-labeling and generative NLP tasks.
Summary updated