TL;DR
MultiOOD introduces the first comprehensive benchmark for multimodal out-of-distribution (OOD) detection, demonstrating that while simply adding modalities helps, specialized training is required to unlock their full potential. The authors propose A2D (Agree-to-Disagree) and NP-Mix, which leverage prediction discrepancies between modalities to significantly improve OOD detection across diverse safety-critical tasks.
What Problem Does This Paper Address?
Machine learning models often fail when they encounter data that differs significantly from their training set—a problem known as Out-of-Distribution (OOD) detection. While vital for safety-critical fields like autonomous driving, most OOD research has focused on unimodal image data.
Real-world applications are inherently multimodal (e.g., combining video, audio, and LiDAR). Existing unimodal OOD methods, when applied to multimodal data, often fail to exploit the complementary nature of these different information streams. Furthermore, the community lacked a standardized, large-scale benchmark to evaluate multimodal OOD detection.
Methods
The authors introduce two primary technical innovations to improve multimodal OOD detection:
- Agree-to-Disagree (A2D) Training: This is based on the observation of Modality Prediction Discrepancy (MPD). For in-distribution (ID) data, different modalities (like video and audio) usually agree on the correct class. However, for OOD data, they tend to "hallucinate" different incorrect classes. A2D formalizes this by adding a loss term that forces modalities to Agree on the ground-truth class for ID data but encourages them to Disagree on all other classes by maximizing the distance between their softmax predictions.
- Nearest Neighbor Prototype-based Mixup (NP-Mix): Standard outlier synthesis often generates "fake" OOD samples too close to the ID data clusters. NP-Mix generates more effective synthetic outliers by identifying class prototypes (mean embeddings) and mixing a sample from one class with the prototype of its nearest neighbor class. This strategy explores the "broader feature spaces" between ID clusters, creating more challenging training signals for the model.
Datasets
The authors established MultiOOD, the first benchmark specifically for multimodal OOD detection. It includes five video-based action recognition datasets:
- EPIC-Kitchens, HAC, HMDB51, UCF101, and Kinetics-600.
- Modalities: Video, Optical Flow, and Audio.
- Scenarios: It covers both Near-OOD (semantic shifts within the same dataset) and Far-OOD (detecting samples from entirely different datasets).
Key Insights and Ideas
- Multimodal Synergy: Simply fusing video and optical flow modalities substantially improves OOD detection over unimodal baselines, even without advanced training strategies.
- Discrepancy as a Signal: The "disagreement" between modalities is a powerful natural indicator of OOD-ness. As shown in the figure below, ID data shows high probability on a single class across modalities, while OOD data shows scattered, conflicting predictions.
- Amplifying the Signal: A2D doesn't just observe this discrepancy; it actively amplifies it during training, making OOD samples much easier to distinguish at test time.
- Filling the Gaps: NP-Mix is superior to standard Mixup because it targets the "empty" regions of the feature space between known classes, which is exactly where OOD samples are likely to appear.
Key Results
The proposed methods achieved state-of-the-art results across the MultiOOD benchmark:
- UCF101 Dataset: The A2D + NP-Mix approach reduced the False Positive Rate (FPR95) from 32.14% to 10.68% for the ASH method— a massive 21.46% absolute improvement.
- Versatility: The improvements were consistent across various backbones (SlowFast, ResNet, Swin-T) and different OOD scoring functions (MSP, Energy, Mahalanobis, etc.).
- Robustness: The model maintained strong performance even when one modality was missing during inference, outperforming models trained on single modalities.
Limitations and Future Work
While the results are impressive, the authors note that Near-OOD tasks (where OOD samples are semantically very similar to ID classes) and datasets with a very large number of classes still pose significant challenges.
Future research directions include exploring Outlier Exposure—using real-world external datasets during training—to further refine the model's ability to learn the boundary between ID and OOD data. The authors also suggest that their framework could be extended beyond video to other multimodal domains like robotics and medical diagnostics.
Summary updated