ADOPD: Reference-Privileged On-Policy Distillation for MLLM-Based Industrial Anomaly Detection

Jingtai He, Shiyuan Meng, Wenchao Meng, Qinmin YangarXiv 2026 · arXiv

TL;DR

ADOPD introduces a reference-privileged on-policy distillation framework that bakes the benefits of image-comparison directly into an MLLM's parameters during training. By evaluating student rollouts against both matched and mismatched reference images, it trains a query-only model to achieve superior zero-shot anomaly detection without the computational overhead of inference-time reference retrieval.

What Problem Does This Paper Address?

Industrial Anomaly Detection (IAD) requires identifying fine-grained defects or irregularities in complex visual data. While Multimodal Large Language Models (MLLMs) excel at open-set reasoning, their open-domain representations struggle to capture subtle, product-specific anomaly patterns.

Providing a "normal" reference image at inference time (one-shot inference) significantly helps MLLMs by giving them a visual baseline for comparison. However, retrieving and processing a reference image for every single query during deployment introduces substantial computational overhead and latency. The authors address a compelling question: Can the benefits of reference comparison be internalized into the model's parameters during training, allowing the model to maintain high accuracy during zero-shot inference?

Figure 1: Reference-conditioned inference to reference-privileged training. (a) A normal ref- erence improves all four tested MLLM backbones. (b) Reference-privileged off-policy distilla- tion trains the student by minimizing the cross-entropy loss on fixed teacher-generated targets. (c) Reference-privileged on-policy distillation evaluates student-generated prefixes with the reference- aware teacher and updates the student through a policy-gradient objective.

Methods

The authors frame this as a "privileged information" problem, where a strong, reference-aware teacher model supervises a query-only student model during training. However, standard on-policy distillation (OPD) can fail if the teacher relies on language priors or query-image cues rather than actual reference comparison.

To solve this, they propose ADOPD (Reference-Privileged On-Policy Distillation), which separates what the model should learn from how strongly it should learn it:

  1. On-Policy Sampling: The trainable student model samples text trajectories (rollouts) based solely on the query image and inspection question.
  2. Asymmetric Reference Intervention: A frozen teacher model replays and evaluates these exact same student-generated tokens under two distinct reference views:
  3. Matched Reference ($r_{mat}$): A regional crop that preserves the valid query-reference relation.
  4. Mismatched Reference ($r_{mis}$): A regional crop from an opposite anomaly state that deliberately breaks the relation.
  5. Token-Level Direction: The teacher-to-student log-probability ratio under the matched reference defines the token-level advantage, guiding the student on what to learn.
  6. Sequence-Level Calibration: The teacher's likelihood gap between the matched and mismatched views serves as a proxy for how much the teacher actually relied on the reference. This gap calibrates a group-relative sequence weight, determining how strongly the rollout should update the student's parameters.
Overview of the ADOPD framework, illustrating token-level advantage calculation and contrastive reference weighting.

Datasets

  • Training: The authors constructed 6,000 paired training records from Real-IAD, covering 30 product categories and 5 camera views. Each defective image was paired with a normal counterfactual image to generate the matched and mismatched regional crops for the teacher.
  • Evaluation: Models were evaluated on the comprehensive MMAD benchmark, which aggregates four major industrial anomaly detection datasets: MVTec-AD, VisA, MVTec-LOCO, and GoodsAD.

Key Insights and Ideas

  • The "Control Group" for Teachers: MLLM teachers can easily hallucinate or rely on language shortcuts. By introducing a mismatched reference as a control, ADOPD effectively audits the teacher: if the teacher gives a response high praise regardless of whether the reference is right or wrong, the framework discounts that feedback.
  • Fixing Exposure Bias: Unlike standard Supervised Fine-Tuning (SFT)—which is an off-policy method training models on fixed teacher targets—ADOPD's on-policy nature ensures the student model is evaluated on its own generation trajectories, leading to better generalization at inference time.
  • Granularity Matters: The authors discovered that supplying full-image references to the teacher was less effective than supplying localized regional crops. Constraining the teacher's privileged view to anomaly-relevant regions produces a much cleaner distillation signal.

Key Results

  • State-of-the-Art Zero-Shot Performance: ADOPD achieved an average accuracy of 77.31% on the MMAD benchmark under zero-shot inference, outperforming all open-source and domain-specific baselines.
  • Beating the One-Shot Baseline: ADOPD improved its Qwen3-VL-4B backbone by 6.14 percentage points and actually outperformed the backbone's one-shot setting by 2.64 points, despite not having access to any reference images at inference time.
  • Targeted Improvements: The performance gains were heavily concentrated in comparison-sensitive tasks like anomaly discrimination (+12.72 points) and defect localization (+2.49 points), proving that the model successfully internalized a fine-grained inspection strategy.

Limitations and Future Work

The primary limitation of ADOPD is its heavy reliance on carefully paired training images and precise defect annotations to construct the matched and mismatched regional crops for the teacher. The authors note that future work will explore moving away from these strict requirements by utilizing weaker or automatically generated privileged references to scale up training data more efficiently.

Summary updated

Abstract

Industrial anomaly detection (IAD) requires identifying fine-grained deviations from normal visual patterns. Multimodal large language models (MLLMs) can improve recognition accuracy by comparing query images with references at inference time, but these benefits rely on additional retrieval and processing. We investigate whether the benefits of reference comparison can instead be internalized in the model parameters. Access to references during training allows a reference-aware teacher to supervise a query-only student. However, the teacher may favor plausible responses based on query cues or language priors rather than valid visual information. We propose ADOPD, a reference-privileged on-policy distillation framework. The teacher evaluates student-generated rollouts under matched and mismatched references. The matched-reference teacher-to-student log-ratio defines the token-level learning direction, specifying what the student should learn. The likelihood gap between the two reference views estimates reference-specific support and calibrates the sequence-level weight. ADOPD achieves 77.31% average accuracy on the MMAD benchmark under zero-shot inference, improving the Qwen3-VL-4B backbone by 6.14 points and outperforming its one-shot setting by 2.64 points. Experiments show that ADOPD learns a fine-grained anomaly inspection strategy from reference comparison. The project will be available at https://github.com/withTai/ADOPD.
BibTeX
@misc{he2026adopd,
  author = {Jingtai He and Shiyuan Meng and Wenchao Meng and Qinmin Yang},
  title = {{ADOPD: Reference-Privileged On-Policy Distillation for MLLM-Based Industrial Anomaly Detection}},
  year = {2026},
  eprint = {2608.09789},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2608.09789},
}
Figure