TL;DR
This paper introduces WOOD, a novel framework for detecting out-of-distribution (OOD) samples in multi-modal data (like images and text). Unlike previous methods that focus on a single type of anomaly, WOOD can simultaneously identify OOD samples arising from various real-world scenarios, such as misaligned data, new domains, or sensor noise, achieving state-of-the-art performance.
What Problem Does This Paper Address?
Ensuring the safety and reliability of machine learning (ML) systems is paramount, especially in critical applications like autonomous driving. A key challenge is Out-of-Distribution (OOD) detection: identifying when an ML system encounters data that is significantly different from what it was trained on. While much research has focused on single-modal OOD detection (e.g., just images or just text), real-world systems often rely on multiple data sources (modalities). Existing multi-modal OOD methods are often limited to a single anomaly scenario, typically where both an image and its corresponding text description are from a new domain. However, practical systems face a wider range of OOD issues, including sensor malfunctions (leading to noisy data), incorrect pairings of data (e.g., an image of a road with text describing a bird), or entirely new environments. This paper addresses the need for a general-purpose multi-modal OOD detection framework capable of handling these diverse anomaly types simultaneously.
Methods
The proposed framework, named WOOD (Weakly-supervised Multi-modal OOD Detection), combines two core components:
- Contrastive Learning Module: This module leverages a pre-trained vision-language model (like CLIP) to learn representations for image-text pairs. The key innovation here is the use of a Hinge loss to explicitly maximize the similarity between in-distribution (ID) pairs while minimizing the similarity for OOD pairs. This helps to create a clearer separation in the learned feature space.
- Binary Classifier Module: This module uses a standard binary classifier to predict the probability of a sample being OOD. To improve its performance, it incorporates a Feature Sparsity Regularizer. This regularizer helps to select and integrate the most informative features from both modalities, preventing one modality from dominating the classification.
A novel scoring metric is introduced to fuse the outputs of both the contrastive learning module and the binary classifier. This metric identifies a sample as OOD only if both components indicate it as such, leading to more robust detection. The framework is trained in a weakly-supervised manner, meaning it uses a small percentage of labeled OOD samples alongside the in-distribution data.
Datasets
The authors evaluated their framework on three real-world datasets:
- COCO: A large-scale object detection dataset with images and captions.
- CUB-200: A dataset of bird images with textual descriptions.
- MIMIC-CXR: A dataset of chest X-ray images with associated textual reports.
For each dataset, they generated three distinct OOD scenarios:
- Scenario 1 (Unaligned Pairs): Images and text descriptions are deliberately mismatched (e.g., an image of a road with text about a bird).
- Scenario 2 (New Domain): OOD samples are drawn from a different dataset entirely (e.g., using Google Conceptual Captions for COCO experiments).
- Scenario 3 (Noisy Data): In-distribution images are corrupted with Gaussian noise (e.g., blur) while their text remains correct.
Key Insights and Ideas
- Complementary Strengths: The paper highlights that contrastive learning and binary classification approaches have complementary strengths for OOD detection. Contrastive learning excels at identifying certain types of anomalies (like misaligned data or noisy images), while binary classifiers are better at others (like samples from entirely new domains). Combining them is crucial for a general-purpose solution.
- Hinge Loss for Separation: Using Hinge loss in the contrastive learning objective is shown to be effective in creating a more distinct separation between in-distribution and out-of-distribution representations, which is key for accurate detection.
- Feature Sparsity for Fusion: The Feature Sparsity Regularizer is a clever way to ensure that the binary classifier effectively utilizes important information from both modalities, rather than being biased by one.
- Unified Scoring for Robustness: The proposed unified scoring metric, which requires agreement from both the contrastive and classification components, significantly improves robustness by reducing false positives.
Key Results
The WOOD framework demonstrated state-of-the-art performance across all three datasets and all three OOD scenarios.
- On the CUB-200 dataset, WOOD achieved an overall accuracy of 89.2%, significantly outperforming baselines like MCM-OOD (51.4%) and CLIP-Energy (33.5%).
- Similar strong results were observed on MIMIC-CXR (overall accuracy 91.0%) and COCO (overall accuracy 98.6%).
- Crucially, WOOD was able to simultaneously detect OOD samples from all three scenarios, a capability that existing methods lacked. For instance, while some baselines performed well on specific scenarios, they failed on others.
Limitations and Future Work
- Recall in Scenario 1: The paper notes that recall for Scenario 1 (unaligned pairs) is not always very high. This is attributed to test samples having very similar images to training data but slightly different textual descriptions, making them challenging to distinguish.
- Hyperparameter Tuning: The performance is sensitive to the weighting parameter ($\lambda$) between the contrastive loss and the binary classifier loss, requiring careful tuning for different datasets (e.g., $\lambda=0.8$ for CUB-200/COCO, $\lambda=0.2$ for MIMIC-CXR).
- Future Work: The authors suggest that their framework's ability to handle multiple OOD scenarios simultaneously opens doors for more robust and reliable ML systems in real-world applications. Further exploration into more complex and diverse OOD scenarios could be a natural next step.
Summary updated