Abstract
Cross-modal anomaly detection identifies data instances that appear normal in isolation but exhibit inconsistent patterns across different modalities. We propose Deep Structured Cross-Modal Anomaly Detection (CMAD), a framework that leverages deep neural networks to project diverse data types into a shared latent space. By capturing complex, nonlinear correlations that traditional linear models miss, CMAD significantly outperforms state-of-the-art baselines on multi-modal datasets like MNIST and RGB-D.
Introduction
Most traditional anomaly detection methods focus on single-source data. However, real-world scenarios—such as fraud detection or multi-sensor monitoring—increasingly involve multi-modal data like images, text, and sensor readings. A "cross-modal anomaly" occurs when individual components are unremarkable, but their combination is inconsistent. For instance, a user’s profile and transaction history might both look "normal" separately, but their mismatch could signal potential fraud.
Existing techniques like Canonical Correlation Analysis (CCA) or Partial Least Squares (PLS) rely on linear mappings. These "shallow" models struggle to capture the highly complex, nonlinear relationships inherent in modern datasets, leading to poor detection performance when data distributions are complicated.
The CMAD Framework
The Deep Structured Cross-Modal Anomaly Detection (CMAD) framework consists of three main stages:
- Feature Extraction & Projection: Modality-specific deep neural networks (e.g., CNNs for images and fully connected networks for text) extract high-level features, which are then projected into a shared, low-dimensional consensus latent feature space.
- The "Pull and Push" Mechanism: During training, the model employs a loss function designed to "pull" projections of consistent modalities together and "push" inconsistent ones further apart using negative sampling and margin-based loss.
- Anomaly Detection: To identify anomalies, the model measures the cosine similarity between projected features. If similarity falls below a pre-defined threshold (ε), the instance is flagged.
Experimental Evaluation
CMAD was evaluated on two primary multi-modal datasets:
- MNIST: 60,000 handwritten digit images with synthetic text tags (embedded via Word2Vec/GloVe).
- RGB-D: 960 pairs of RGB images and corresponding depth maps.
Since these datasets lack natural cross-modal anomalies, the authors used negative sampling to inject mismatched pairs for testing.
Key Insights and Results
CMAD demonstrated a substantial performance leap over traditional baselines. On MNIST, it achieved an accuracy of 99.21%, a precision of 99.71%, and a recall of 98.75%, compared to ~66% accuracy for linear CCA. On the RGB-D dataset, CMAD maintained 95.12% accuracy, significantly outperforming PLS (65.80%) and HOAD (51.37%).
A core finding is that cross-modal correlations are rarely linear; deep encoders are essential for learning a "common language" between data types. Visualizations of the shared latent space show that consistent pairs naturally cluster together regardless of their original modality.
Conclusion and Future Work
CMAD provides a robust method for detecting relational mismatches in multi-modal data. Future research will focus on extending the framework to sequential or networked data (like video or social media graphs) and developing methods for handling high-velocity streaming data in real-time. Additionally, reducing the reliance on supervised class labels during training remains a key objective for purely unsupervised settings.
Summary updated