DMix: Adaptive Distance-aware Interpolative Mixup

Ramit Sawhney, Megh Thakkar, Shrey Pandit, Ritesh Soun, Di Jin, Diyi Yang, Lucie Flek

Abstract

Data augmentation techniques like Mixup have become standard for regularizing deep learning models. However, traditional Mixup often relies on random sampling and Euclidean distance, which may not capture the inherent hierarchical structures of natural language. We present DMix, an adaptive distance-aware interpolative Mixup strategy that leverages hyperbolic geometry to identify diverse samples and employs a learnable matrix to optimize interpolation ratios. Our experiments across eight multilingual datasets demonstrate that DMix significantly outperforms existing baselines in both performance and training efficiency.

1. Introduction

Deep learning models often require large amounts of labeled data to generalize effectively. Mixup, a popular data augmentation technique, generates synthetic training examples by linearly interpolating pairs of samples and their labels. Despite its success, standard Mixup suffers from two primary limitations: it selects pairs randomly without considering their semantic relationship, and it typically operates in Euclidean space, which is suboptimal for representing the hierarchical nature of linguistic data.

2. Methodology

DMix addresses these limitations through three core components: hyperbolic embeddings, diversity-based sample selection, and adaptive interpolation.

2.1 Hyperbolic Geometry for Language

Unlike Euclidean space, hyperbolic space (modeled here as a Poincaré ball) has a volume that grows exponentially with its radius. This property makes it exceptionally well-suited for embedding hierarchical structures like trees or language manifolds. DMix calculates distances between sentence embeddings in this hyperbolic space to better capture semantic diversity.

2.2 Diversity-Aware Selection and Adaptive Interpolation

Instead of random pairing, DMix uses a distance-aware strategy. For a given input $x_i$, it identifies a set of diverse candidates $x_j$ whose hyperbolic distance $D_h(x_i, x_j)$ exceeds a predefined threshold $ au$. Furthermore, DMix introduces a learnable matrix $M$ that determines the interpolation weights based on these distances. This matrix is updated via backpropagation, allowing the model to adaptively learn the optimal mixing strategy for a given task.

The DMix architecture integrates hyperbolic distance measurements and a learnable matrix M to optimize the Mixup process.

3. Experimental Setup

We evaluated DMix on eight datasets across four languages: English (TRAC, TREC-Coarse, TREC-Fine, CoLA, SST-2), Arabic (AHS), Turkish (TTC), and Hindi (HASOC). We utilized BERT for English tasks and mBERT for multilingual tasks as our base encoders.

4. Results and Discussion

DMix consistently outperformed all baseline methods, including standard interpolative Mixup (TMix) and variants using Euclidean distance.

4.1 Performance Gains

As shown in the comparative analysis, DMix achieved state-of-the-art F1 scores across all benchmarks. Notably, on the CoLA dataset, DMix improved the F1 score from 85.30 (TMix) to 95.94. The comparison between Euclidean DMix (Euc-DMix) and hyperbolic DMix confirms that the choice of geometry is critical for linguistic tasks.

Performance comparison of DMix against various baselines across eight datasets. Bold values indicate best performance.

4.2 Training Efficiency

Beyond raw performance, DMix exhibits superior convergence properties. On datasets like TRAC and HASOC, DMix reached target accuracy levels in approximately one-third of the iterations required by standard Mixup. This efficiency is attributed to the diversity-aware selection process, which prioritizes informative synthetic samples early in training.

5. Conclusion

DMix represents a significant advancement in interpolative data augmentation. By combining the structural advantages of hyperbolic geometry with an adaptive, learnable mixing strategy, it provides a robust and efficient framework for sentence classification. Future work will explore the application of DMix to other modalities, such as computer vision and speech, where hierarchical relationships are also prevalent.

Summary updated

BibTeX
@inproceedings{sawhney2022dmix,
  author = {Ramit Sawhney and Megh Thakkar and Shrey Pandit and Ritesh Soun and Di Jin and Diyi Yang and Lucie Flek},
  title = {{DMix: Adaptive Distance-aware Interpolative Mixup}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2022},
  doi = {10.18653/v1/2022.acl-short.67},
}
Figure