MultiOOD: Scaling Out-of-Distribution Detection for Multiple Modalities

Hao Dong, Yue Zhao, Eleni Chatzi, Olga Fink

TL;DR

MultiOOD introduces the first comprehensive benchmark for multimodal out-of-distribution (OOD) detection, demonstrating that while simply adding modalities helps, specialized training is required to unlock their full potential. The authors propose A2D (Agree-to-Disagree) and NP-Mix, which leverage prediction discrepancies between modalities to significantly improve OOD detection across diverse safety-critical tasks.

What Problem Does This Paper Address?

Machine learning models often fail when they encounter data that differs significantly from their training set—a problem known as Out-of-Distribution (OOD) detection. While vital for safety-critical fields like autonomous driving, most OOD research has focused on unimodal image data.

Real-world applications are inherently multimodal (e.g., combining video, audio, and LiDAR). Existing unimodal OOD methods, when applied to multimodal data, often fail to exploit the complementary nature of these different information streams. Furthermore, the community lacked a standardized, large-scale benchmark to evaluate multimodal OOD detection.

Methods

The authors introduce two primary technical innovations to improve multimodal OOD detection:

  1. Agree-to-Disagree (A2D) Training: This is based on the observation of Modality Prediction Discrepancy (MPD). For in-distribution (ID) data, different modalities (like video and audio) usually agree on the correct class. However, for OOD data, they tend to "hallucinate" different incorrect classes. A2D formalizes this by adding a loss term that forces modalities to Agree on the ground-truth class for ID data but encourages them to Disagree on all other classes by maximizing the distance between their softmax predictions.
  2. Nearest Neighbor Prototype-based Mixup (NP-Mix): Standard outlier synthesis often generates "fake" OOD samples too close to the ID data clusters. NP-Mix generates more effective synthetic outliers by identifying class prototypes (mean embeddings) and mixing a sample from one class with the prototype of its nearest neighbor class. This strategy explores the "broader feature spaces" between ID clusters, creating more challenging training signals for the model.
The A2D and NP-Mix framework leverages multiple modalities to identify OOD samples.

Datasets

The authors established MultiOOD, the first benchmark specifically for multimodal OOD detection. It includes five video-based action recognition datasets:

  • EPIC-Kitchens, HAC, HMDB51, UCF101, and Kinetics-600.
  • Modalities: Video, Optical Flow, and Audio.
  • Scenarios: It covers both Near-OOD (semantic shifts within the same dataset) and Far-OOD (detecting samples from entirely different datasets).
Figure 2: An overview of MultiOOD Benchmark.

Key Insights and Ideas

  • Multimodal Synergy: Simply fusing video and optical flow modalities substantially improves OOD detection over unimodal baselines, even without advanced training strategies.
  • Discrepancy as a Signal: The "disagreement" between modalities is a powerful natural indicator of OOD-ness. As shown in the figure below, ID data shows high probability on a single class across modalities, while OOD data shows scattered, conflicting predictions.
  • Amplifying the Signal: A2D doesn't just observe this discrepancy; it actively amplifies it during training, making OOD samples much easier to distinguish at test time.
  • Filling the Gaps: NP-Mix is superior to standard Mixup because it targets the "empty" regions of the feature space between known classes, which is exactly where OOD samples are likely to appear.
Softmax outputs for ID data (left) show high agreement, while OOD data (right) shows significant discrepancy between modalities.

Key Results

The proposed methods achieved state-of-the-art results across the MultiOOD benchmark:

  • UCF101 Dataset: The A2D + NP-Mix approach reduced the False Positive Rate (FPR95) from 32.14% to 10.68% for the ASH method— a massive 21.46% absolute improvement.
  • Versatility: The improvements were consistent across various backbones (SlowFast, ResNet, Swin-T) and different OOD scoring functions (MSP, Energy, Mahalanobis, etc.).
  • Robustness: The model maintained strong performance even when one modality was missing during inference, outperforming models trained on single modalities.
Figure 1: The FPR95 (lower is better) and AUROC (higher is better) on HMDB51 dataset across various modalities. Multimodal OOD substantially improves unimodal OOD w/o bells and whistles.

Limitations and Future Work

While the results are impressive, the authors note that Near-OOD tasks (where OOD samples are semantically very similar to ID classes) and datasets with a very large number of classes still pose significant challenges.

Future research directions include exploring Outlier Exposure—using real-world external datasets during training—to further refine the model's ability to learn the boundary between ID and OOD data. The authors also suggest that their framework could be extended beyond video to other multimodal domains like robotics and medical diagnostics.

Summary updated

Abstract

Detecting out-of-distribution (OOD) samples is important for deploying machine learning models in safety-critical applications such as autonomous driving and robot-assisted surgery. Existing research has mainly focused on unimodal scenarios on image data. However, real-world applications are inherently multimodal, which makes it essential to leverage information from multiple modalities to enhance the efficacy of OOD detection. To establish a foundation for more realistic Multimodal OOD Detection, we introduce the first-of-its-kind benchmark, MultiOOD, characterized by diverse dataset sizes and varying modality combinations. We first evaluate existing unimodal OOD detection algorithms on MultiOOD, observing that the mere inclusion of additional modalities yields substantial improvements. This underscores the importance of utilizing multiple modalities for OOD detection. Based on the observation of Modality Prediction Discrepancy between in-distribution (ID) and OOD data, and its strong correlation with OOD performance, we propose the Agree-to-Disagree (A2D) algorithm to encourage such discrepancy during training. Moreover, we introduce a novel outlier synthesis method, NP-Mix, which explores broader feature spaces by leveraging the information from nearest neighbor classes and complements A2D to strengthen OOD detection performance. Extensive experiments on MultiOOD demonstrate that training with A2D and NP-Mix improves existing OOD detection algorithms by a large margin. Our source code and MultiOOD benchmark are available at https://github.com/donghao51/MultiOOD.
BibTeX
@inproceedings{dong2024multiood,
  author = {Hao Dong and Yue Zhao and Eleni Chatzi and Olga Fink},
  title = {{MultiOOD: Scaling Out-of-Distribution Detection for Multiple Modalities}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2024},
  url = {https://openreview.net/forum?id=A5pabdZp2F},
}
Figure