Co-Progression Knowledge Distillation with Knowledge Prototype for Industrial Anomaly Detection

Bokang Yang, Zhe Zhang, Jie MaAAAI 2025 · DOI

TL;DR

This paper introduces a novel "Co-Progression Knowledge Distillation" (CPKD) framework for unsupervised industrial anomaly detection. By enabling bidirectional learning between teacher and student models and introducing a "Knowledge Prototype" to regulate the teacher's learning, CPKD achieves state-of-the-art performance on the MVTec dataset, outperforming existing methods by striking a better balance between learning new information and retaining core knowledge.

What Problem Does This Paper Address?

Industrial anomaly detection is crucial for quality control, but obtaining labeled defective samples is often difficult. Unsupervised methods, which learn from normal data, are thus highly valuable. A common approach is knowledge distillation, where a student model learns from a pre-trained teacher model. However, existing methods often suffer from "excessive specialization" (student learns abnormal features) or "inadequate generalization" (student fails to fully capture normal patterns), leading to reduced detection accuracy. This paper aims to overcome these limitations by proposing a more dynamic and robust knowledge distillation process.

Methods

The core innovation is the Co-Progression Knowledge Distillation (CPKD) framework, which introduces two key components:

  • Co-Progression (CP): Unlike traditional methods where the teacher is static, CPKD allows for bidirectional learning. The student learns from the teacher, but the teacher also learns and adapts from the student's evolving representations. This dynamic interaction aims to accelerate convergence and improve efficiency.
  • Knowledge Prototype (KP): To prevent the teacher from deviating too far from its core knowledge of normal patterns during the co-progression process (i.e., to avoid degradation or overfitting), a "Knowledge Prototype" is introduced. This prototype is generated by a separate, frozen "teaching monitor" model trained on normal samples. The teacher model is then guided by this prototype, ensuring it maintains stability and focuses on learning normal patterns.

The training process involves:

  1. Prototype Generation: A teaching monitor model processes normal images to extract features, which are then aggregated and subsampled (using coreset subsampling) to form the Knowledge Prototype.
  2. Co-Progression Distillation: The student model learns from the teacher's multi-level outputs via a knowledge distillation loss. Simultaneously, the teacher model updates its parameters based on both the student's output (to adapt) and its alignment with the Knowledge Prototype (to regulate).

Datasets

The paper uses the following datasets for evaluation:

  • MVTec AD: A widely used benchmark for anomaly detection, comprising 15 object categories with normal and defective samples. It's crucial for evaluating performance on both texture and object anomalies.
  • VisA: A more challenging dataset with 12 classes of industrial products, known for its complex textures and subtle anomalies, making it a good testbed for robust detection.

Key Insights and Ideas

  • Bidirectional Learning is Key: The paper highlights that simply having a student learn from a teacher is insufficient. Allowing the teacher to adapt based on the student's unique perspective is crucial for mutual improvement and better anomaly detection.
  • Regulation is Essential: While bidirectional learning is powerful, it needs a stabilizing force. The Knowledge Prototype acts as a "regulatory mechanism," preventing the teacher from "forgetting" what normal looks like and thus mitigating overfitting in the student.
  • Balancing Act: The CPKD framework successfully balances acquiring new knowledge (through co-progression) with preserving core competencies (through the knowledge prototype), leading to improved adaptability and generalization.
  • Dynamic Teacher: The concept of a teacher model that can dynamically adjust its parameters based on student interaction is a significant departure from traditional knowledge distillation paradigms.

Key Results

  • State-of-the-Art Performance: The proposed CPKP framework achieves state-of-the-art (SOTA) performance on the MVTec AD dataset, with an average image-level AUC-ROC of 99.55% and pixel-level AUC-ROC of 98.01%.
  • Competitive on VisA: CPKP(RD) demonstrates competitive results on the VisA dataset, achieving the second-best average image-level AUC-ROC (96.99%) and third-best pixel-level AUC-ROC (98.54%).
  • Ablation Study Validation: An ablation study confirms the effectiveness of both Co-Progression and Knowledge Prototype components. Individually, they offer some benefits, but their combination yields significant synergistic improvements, outperforming baseline methods and individual component approaches.
  • Hyperparameter Sensitivity: The hyperparameter 'α', controlling the weight of the prototype loss, shows that higher values (closer to 1) generally lead to better performance, indicating the importance of the prototype in guiding the teacher.

Limitations and Future Work

  • Hyperparameter Tuning: While the paper explores the impact of 'α', further investigation into optimal hyperparameter settings for various scenarios might be beneficial.
  • Computational Cost: The introduction of a teaching monitor and bidirectional learning might increase computational complexity, though the paper claims improved efficiency. Further analysis on real-time applicability in industrial settings could be explored.
  • Generalization to Other Domains: While successful on industrial datasets, exploring the framework's effectiveness in other anomaly detection domains (e.g., medical imaging, cybersecurity) could be a promising future direction.
  • Understanding Teacher Adaptation: Deeper analysis into how the teacher model adapts and what specific insights it gains from the student could provide further theoretical understanding.

Summary updated

BibTeX
@inproceedings{yang2025co,
  author = {Bokang Yang and Zhe Zhang and Jie Ma},
  title = {{Co-Progression Knowledge Distillation with Knowledge Prototype for Industrial Anomaly Detection}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i12.33419},
}
Figure