TL;DR
This paper introduces the Multimodal Task Representation Memory Bank (MTRMB) to tackle catastrophic forgetting in unsupervised continuous anomaly detection (UCAD). By leveraging multimodal prompts and refined structure-based contrastive learning, MTRMB effectively retains knowledge across tasks, achieving state-of-the-art performance with significantly reduced forgetting.
What Problem Does This Paper Address?
Industrial anomaly detection often requires systems to continuously learn from new data without forgetting previously acquired knowledge. This is particularly challenging in unsupervised settings where explicit labels are scarce. Existing methods struggle with "catastrophic forgetting," where learning a new task erases knowledge of old ones, leading to poor performance. Furthermore, relying solely on image features can result in incomplete representations, hindering the ability to distinguish between redundant and complementary information across different tasks. This paper addresses these limitations by proposing a novel approach for unsupervised continuous anomaly detection that effectively manages knowledge across multiple tasks.
Methods
The core of the proposed solution is the Multimodal Task Representation Memory Bank (MTRMB), built upon two key innovations:
- Key-Prompt-Multimodal Knowledge (KPMK) Mechanism: This mechanism uses concise "key prompts" to guide the interaction between text (BERT) and image (ViT) features. This cross-modal interaction helps build a more comprehensive and robust representation of each task's knowledge. The KPMK mechanism stores task-specific keys and prompts, enabling efficient task matching and knowledge transfer during inference.
- Refined Structure-based Contrastive Learning (RSCL): To create more compact and discriminative feature representations, RSCL leverages pre-trained models like Grounding DINO and SAM. These models generate precise segmentation masks, allowing the RSCL to pull features from similar structural regions closer together while pushing dissimilar ones apart. This process refines the task representation memory bank, making it more robust and less prone to forgetting.
During inference, MTRMB queries the task key to retrieve the corresponding prompt, adapts the model to the current task domain, and then compares the extracted image features with the stored normal knowledge to detect anomalies.
Datasets
The proposed MTRMB method was evaluated on two widely used datasets for industrial image anomaly detection:
- MVTec AD [1]: A standard dataset for unsupervised anomaly detection.
- VisA [2]: A larger dataset for real-world industrial anomaly detection with pixel-level annotations.
The paper does not introduce a new benchmark dataset but rather focuses on evaluating existing methods and their proposed approach on these established benchmarks.
Key Insights and Ideas
- Multimodal Fusion is Key for Continual Learning: The paper strongly suggests that integrating information from multiple modalities (text and image) is crucial for overcoming catastrophic forgetting in unsupervised anomaly detection. Textual prompts provide valuable contextual information that complements visual features.
- Structured Representations Combat Forgetting: By using segmentation masks to guide feature learning (RSCL), the model can create more compact and discriminative representations. This structural awareness helps the model retain knowledge about different object parts and their relationships, making it less susceptible to forgetting.
- Prompt Engineering Matters: The study highlights that the length and nature of prompts can significantly impact the model's performance in continuous learning scenarios, suggesting that careful prompt design is critical for optimal results.
- Memory Bank for Knowledge Retention: The concept of a task representation memory bank, enhanced by multimodal information and structural refinement, proves effective in storing and recalling knowledge across diverse tasks.
Key Results
The MTRMB method demonstrated superior performance on both MVTec AD and VisA datasets.
- Average Detection Accuracy: Achieved an average detection accuracy of 0.921 on MVTec AD.
- Lowest Forgetting Rate: Significantly outperformed state-of-the-art methods in terms of preventing catastrophic forgetting.
- Quantitative Superiority: Consistently achieved higher Image AUROC and Pixel AUROC/AP scores compared to other advanced methods, including DNE and UCAD, especially in the continuous learning scenario. For instance, on MVTec AD, the proposed method showed an Image AUROC that was 0.071 and 0.015 higher than DNE and UCAD, respectively.
Qualitative results (visualizations of anomaly detection heatmaps) also showed more accurate anomaly localization and fewer false positives compared to other methods.
Limitations and Future Work
- Prompt Hyperparameter Sensitivity: The paper acknowledges that the performance is influenced by prompt hyperparameters, and while they found an optimal range (prompt length of 5), further exploration into prompt design could yield even better results.
- Computational Cost: While the method aims to be efficient, the use of large multimodal models like BERT and ViT can still incur significant computational costs.
- Future Work: The authors plan to open-source their code. They also suggest that further optimization of prompt hyperparameters is an important direction for future work to enhance continuous learning capabilities.
Summary updated