Deep Structured Cross-Modal Anomaly Detection

Yuening Li, Ninghao Liu, Jundong Li, Mengnan Du, Xia Hu

Abstract

Cross-modal anomaly detection identifies data instances that appear normal in isolation but exhibit inconsistent patterns across different modalities. We propose Deep Structured Cross-Modal Anomaly Detection (CMAD), a framework that leverages deep neural networks to project diverse data types into a shared latent space. By capturing complex, nonlinear correlations that traditional linear models miss, CMAD significantly outperforms state-of-the-art baselines on multi-modal datasets like MNIST and RGB-D.

Introduction

Most traditional anomaly detection methods focus on single-source data. However, real-world scenarios—such as fraud detection or multi-sensor monitoring—increasingly involve multi-modal data like images, text, and sensor readings. A "cross-modal anomaly" occurs when individual components are unremarkable, but their combination is inconsistent. For instance, a user’s profile and transaction history might both look "normal" separately, but their mismatch could signal potential fraud.

Existing techniques like Canonical Correlation Analysis (CCA) or Partial Least Squares (PLS) rely on linear mappings. These "shallow" models struggle to capture the highly complex, nonlinear relationships inherent in modern datasets, leading to poor detection performance when data distributions are complicated.

The CMAD Framework

The Deep Structured Cross-Modal Anomaly Detection (CMAD) framework consists of three main stages:

  1. Feature Extraction & Projection: Modality-specific deep neural networks (e.g., CNNs for images and fully connected networks for text) extract high-level features, which are then projected into a shared, low-dimensional consensus latent feature space.
  2. The "Pull and Push" Mechanism: During training, the model employs a loss function designed to "pull" projections of consistent modalities together and "push" inconsistent ones further apart using negative sampling and margin-based loss.
  3. Anomaly Detection: To identify anomalies, the model measures the cosine similarity between projected features. If similarity falls below a pre-defined threshold (ε), the instance is flagged.
The Pull and Push mechanism projects different modalities into a shared latent space to identify inconsistencies.

Experimental Evaluation

CMAD was evaluated on two primary multi-modal datasets:

  • MNIST: 60,000 handwritten digit images with synthetic text tags (embedded via Word2Vec/GloVe).
  • RGB-D: 960 pairs of RGB images and corresponding depth maps.

Since these datasets lack natural cross-modal anomalies, the authors used negative sampling to inject mismatched pairs for testing.

Key Insights and Results

CMAD demonstrated a substantial performance leap over traditional baselines. On MNIST, it achieved an accuracy of 99.21%, a precision of 99.71%, and a recall of 98.75%, compared to ~66% accuracy for linear CCA. On the RGB-D dataset, CMAD maintained 95.12% accuracy, significantly outperforming PLS (65.80%) and HOAD (51.37%).

A core finding is that cross-modal correlations are rarely linear; deep encoders are essential for learning a "common language" between data types. Visualizations of the shared latent space show that consistent pairs naturally cluster together regardless of their original modality.

Visualization of the shared latent space where different digit classes form distinct clusters across modalities.

Conclusion and Future Work

CMAD provides a robust method for detecting relational mismatches in multi-modal data. Future research will focus on extending the framework to sequential or networked data (like video or social media graphs) and developing methods for handling high-velocity streaming data in real-time. Additionally, reducing the reliance on supervised class labels during training remains a key objective for purely unsupervised settings.

Summary updated

Abstract

Anomaly detection is a fundamental problem in data mining field with many real-world applications. A vast majority of existing anomaly detection methods predominately focused on data collected from a single source. In real-world applications, instances often have multiple types of features, such as images (ID photos, finger prints) and texts (bank transaction histories, user online social media posts), resulting in the so-called multi-modal data. In this paper, we focus on identifying anomalies whose patterns are disparate across different modalities, i.e., cross-modal anomalies. Some of the data instances within a multi-modal context are often not anomalous when they are viewed separately in each individual modality, but contains inconsistent patterns when multiple sources are jointly considered. The existence of multi-modal data in many real-world scenarios brings both opportunities and challenges to the canonical task of anomaly detection. On the one hand, in multimodal data, information of different modalities may complement each other in improving the detection performance. On the other hand, complicated distributions across different modalities call for a principled framework to characterize their inherent and complex correlations, which is often difficult to capture with conventional linear models. To this end, we propose a novel deep structured anomaly detection framework to identify the cross-modal anomalies embedded in the data. Experiments on real-world datasets demonstrate the effectiveness of the proposed framework comparing with the state-of-the-art.
BibTeX
@inproceedings{li2019deep,
  author = {Yuening Li and Ninghao Liu and Jundong Li and Mengnan Du and Xia Hu},
  title = {{Deep Structured Cross-Modal Anomaly Detection}},
  booktitle = {IJCNN},
  year = {2019},
  url = {https://openreview.net/forum?id=xJ0H_7C_GDW7},
}
Figure