MIDAS: Misalignment-based Data Augmentation Strategy for Imbalanced Multimodal Learning

Seonghyeon Hwang, Soyoung Choi, Steven Euijong Whang

Introduction: The Challenge of Modality Imbalance

In multimodal learning, models are designed to integrate diverse data streams—such as video, audio, and text—to perform complex tasks like action recognition or sentiment analysis. However, these models often suffer from modality imbalance, a phenomenon where the network over-relies on a single "dominant" modality that is easier to optimize or more informative during training. This "shortcut" learning leads to the neglect of "weak" modalities, resulting in brittle models that fail when the dominant signal is noisy or unavailable.

The MIDAS Strategy

To address this, the authors introduce MIDAS (Misalignment-based Data Augmentation Strategy). Unlike previous approaches that modify loss functions or mask inputs, MIDAS leverages the data itself to force models to extract information from all available sources.

The MIDAS framework architecture, illustrating misaligned sample generation and the application of confidence-based labels and weighting mechanisms.

The strategy is built on three core components:

  1. Misaligned Sample Generation: MIDAS intentionally creates "synthetic" samples by swapping modalities within a training batch. For instance, it might pair the visual frames of a "running" action with the audio of a "clapping" action. This creates a semantic conflict that prevents the model from relying on a single modality to identify the class.
  2. Confidence-based Labeling: Since misaligned samples contain conflicting information, they cannot be assigned a simple "hard" label. MIDAS employs pre-trained unimodal classifiers to assess the confidence of each modality. It then generates a "soft" label—a weighted distribution—that reflects the relative contribution of each modality to the final prediction.
  3. Two-Tier Weighting Mechanism:
    • Weak-Modality Weighting (WM): This component dynamically increases the loss weight for modalities that the model is currently under-utilizing, effectively forcing the network to pay more attention to the "ignored" signals.
  4. Hard-Sample Weighting (HS): MIDAS identifies "hard" misaligned samples—those where features are semantically similar but belong to different classes—and assigns them higher weights to provide a stronger training signal.

Experimental Setup and Datasets

The effectiveness of MIDAS was validated across a diverse range of multimodal benchmarks:

  • Audio-Visual: Kinetics-Sounds (action recognition) and CREMA-D (emotion recognition).
  • Video-Motion: UCF-101 (RGB frames and optical flow).
  • Image-Text: Food-101 (images and recipes), Sarcasm, and Twitter2015.
  • Trimodal: CMU-MOSI (text, audio, and video for sentiment analysis).

Key Results and Insights

MIDAS consistently outperformed existing state-of-the-art (SOTA) methods across all tested datasets. By forcing the model to resolve semantic conflicts, MIDAS significantly improved the performance of the weaker modalities and narrowed the "modality gap."

Comparison of MIDAS performance (Accuracy and F1 Score) against Joint Training and various state-of-the-art baselines across four major datasets.

Key findings include:

  • Significant Accuracy Gains: On the Kinetics-Sounds dataset, MIDAS achieved an accuracy of 74.88%, a substantial improvement over the 63.92% achieved by standard joint training and the previous best baseline of 71.75%.
  • Enhanced Robustness: The model's ability to handle misaligned data improved drastically, proving that it had learned to extract meaningful features from both modalities rather than relying on shortcuts.
  • Efficiency through Feature-Level Augmentation: By performing swaps at the feature level (after encoders), MIDAS maintains high computational efficiency, avoiding the need to re-process raw high-dimensional inputs.

Conclusion and Future Directions

MIDAS represents a powerful shift in multimodal learning, demonstrating that data augmentation through intentional misalignment can effectively mitigate modality imbalance. While currently optimized for classification, future work may extend this strategy to generative or retrieval tasks. Additionally, exploring automated hyperparameter tuning and scaling to even higher-dimensional multimodal inputs remain promising avenues for research.

Summary updated

BibTeX
@inproceedings{hwang2025midas,
  author = {Seonghyeon Hwang and Soyoung Choi and Steven Euijong Whang},
  title = {{MIDAS: Misalignment-based Data Augmentation Strategy for Imbalanced Multimodal Learning}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2025},
  url = {https://openreview.net/forum?id=YAycrn1Idm},
}
Figure