Abstract
Data augmentation techniques like Mixup have become standard for regularizing deep learning models. However, traditional Mixup often relies on random sampling and Euclidean distance, which may not capture the inherent hierarchical structures of natural language. We present DMix, an adaptive distance-aware interpolative Mixup strategy that leverages hyperbolic geometry to identify diverse samples and employs a learnable matrix to optimize interpolation ratios. Our experiments across eight multilingual datasets demonstrate that DMix significantly outperforms existing baselines in both performance and training efficiency.
1. Introduction
Deep learning models often require large amounts of labeled data to generalize effectively. Mixup, a popular data augmentation technique, generates synthetic training examples by linearly interpolating pairs of samples and their labels. Despite its success, standard Mixup suffers from two primary limitations: it selects pairs randomly without considering their semantic relationship, and it typically operates in Euclidean space, which is suboptimal for representing the hierarchical nature of linguistic data.
2. Methodology
DMix addresses these limitations through three core components: hyperbolic embeddings, diversity-based sample selection, and adaptive interpolation.
2.1 Hyperbolic Geometry for Language
Unlike Euclidean space, hyperbolic space (modeled here as a Poincaré ball) has a volume that grows exponentially with its radius. This property makes it exceptionally well-suited for embedding hierarchical structures like trees or language manifolds. DMix calculates distances between sentence embeddings in this hyperbolic space to better capture semantic diversity.
2.2 Diversity-Aware Selection and Adaptive Interpolation
Instead of random pairing, DMix uses a distance-aware strategy. For a given input $x_i$, it identifies a set of diverse candidates $x_j$ whose hyperbolic distance $D_h(x_i, x_j)$ exceeds a predefined threshold $ au$. Furthermore, DMix introduces a learnable matrix $M$ that determines the interpolation weights based on these distances. This matrix is updated via backpropagation, allowing the model to adaptively learn the optimal mixing strategy for a given task.
3. Experimental Setup
We evaluated DMix on eight datasets across four languages: English (TRAC, TREC-Coarse, TREC-Fine, CoLA, SST-2), Arabic (AHS), Turkish (TTC), and Hindi (HASOC). We utilized BERT for English tasks and mBERT for multilingual tasks as our base encoders.
4. Results and Discussion
DMix consistently outperformed all baseline methods, including standard interpolative Mixup (TMix) and variants using Euclidean distance.
4.1 Performance Gains
As shown in the comparative analysis, DMix achieved state-of-the-art F1 scores across all benchmarks. Notably, on the CoLA dataset, DMix improved the F1 score from 85.30 (TMix) to 95.94. The comparison between Euclidean DMix (Euc-DMix) and hyperbolic DMix confirms that the choice of geometry is critical for linguistic tasks.
4.2 Training Efficiency
Beyond raw performance, DMix exhibits superior convergence properties. On datasets like TRAC and HASOC, DMix reached target accuracy levels in approximately one-third of the iterations required by standard Mixup. This efficiency is attributed to the diversity-aware selection process, which prioritizes informative synthetic samples early in training.
5. Conclusion
DMix represents a significant advancement in interpolative data augmentation. By combining the structural advantages of hyperbolic geometry with an adaptive, learnable mixing strategy, it provides a robust and efficient framework for sentence classification. Future work will explore the application of DMix to other modalities, such as computer vision and speech, where hierarchical relationships are also prevalent.
Summary updated