HypMix: Hyperbolic Interpolative Data Augmentation

Ramit Sawhney, Megh Thakkar, Shivam Agarwal, Di Jin, Diyi Yang, Lucie Flek

Introduction

Data augmentation is a cornerstone of modern deep learning, particularly when labeled data is scarce. A prominent technique is Mixup, which creates synthetic training examples by linearly interpolating between pairs of inputs and their corresponding labels in Euclidean space. However, many data modalities—such as natural language with its hierarchical syntax or audio signals with hyperbolic interference—possess inherent non-Euclidean structures. Forcing these "tree-like" structures into flat Euclidean space can lead to significant vectorial distortions, resulting in suboptimal synthetic samples.

To address this, we present HypMix, a novel data augmentation framework that performs interpolation within hyperbolic space. By leveraging the Poincaré ball model, HypMix captures the hierarchical and complex latent structures of data more effectively than standard Euclidean methods.

Methodology

The core innovation of HypMix lies in moving the interpolation process from the flat Euclidean plane to the curved geometry of the Poincaré ball.

MixH and HypMix

The fundamental operation, MixH (Hyperbolic Mixup), replaces standard linear addition with Möbius operations. As illustrated in Figure 1, the process involves:

  1. Mapping Euclidean hidden states ($h_k$) into hyperbolic space using an exponential map ($\exp_0$).
  2. Performing a weighted Möbius addition and scalar multiplication within the hyperbolic manifold.
  3. Projecting the resulting mixed representation back to Euclidean space via a logarithmic map ($\log_0$).
HypMix Architecture: The diagram shows the flow from input data to hyperbolic interpolation (MixH) within the hidden layers of a base model.

This approach is modality-agnostic and can be applied to the latent representations of any neural network architecture, such as BERT for text or ResNet for vision.

Möbius Gyromidpoint Label Estimation (MGLE)

For semi-supervised learning, the authors introduced MGLE. This technique estimates labels for unlabeled data by generating multiple augmentations of a single sample and calculating their hyperbolic weighted midpoint. This produces a stable "soft label" that respects the underlying geometry of the feature space.

Experimental Setup

HypMix was evaluated across three distinct modalities to prove its versatility:

  • Speech: ESC-10, US8K, and Urdu SER (emotion recognition).
  • Text: AG News, DB Pedia, and Arabic HS (hate speech detection).
  • Vision: CIFAR-10 and CIFAR-100.

The inclusion of low-resource datasets like Urdu SER and Arabic HS was critical to testing the model's performance in data-constrained environments.

Results and Discussion

HypMix consistently outperformed standard Euclidean Mixup (EUCMIX) across all modalities and tasks.

Performance in Low-Resource Settings

The advantages of hyperbolic interpolation are most pronounced when training data is extremely limited. For instance, on the Urdu Speech Emotion dataset with only 5 samples per class, HypMix achieved an error rate of 16.5%, significantly lower than the 18.8% achieved by Euclidean Mixup and the 28.7% of the base model.

Performance Gains: Percentage improvement of HypMix over baselines (SpeechMix and MixText) for Urdu SER and Arabic HS datasets across different sample sizes.

Robustness and Efficiency

Beyond accuracy, HypMix enhances model robustness. Models trained with this method showed a ~6% improvement in resistance to adversarial attacks (e.g., FGSM). Furthermore, despite the mathematical complexity of hyperbolic operations, HypMix is computationally efficient. On modern GPUs, the operations are highly parallelizable, adding only a negligible constant factor to training time compared to standard Mixup.

Conclusion

HypMix demonstrates that the geometry of the augmentation space is a critical factor in model generalization. By aligning the interpolation manifold with the inherent hierarchical structure of the data, HypMix provides a more physically and semantically plausible set of synthetic examples. Future work will explore its application to large-scale benchmarks like GLUE and its potential in multimodal learning environments.

Summary updated

BibTeX
@inproceedings{sawhney2021hypmix,
  author = {Ramit Sawhney and Megh Thakkar and Shivam Agarwal and Di Jin and Diyi Yang and Lucie Flek},
  title = {{HypMix: Hyperbolic Interpolative Data Augmentation}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2021},
  doi = {10.18653/v1/2021.emnlp-main.776},
}
Figure