AnomalyJev: Towards Evaluating Image Anomaly Explanations via Claim-Level Verification

Qizhou Wang, Jiawen Zhu, Guansong Pang, Christopher LeckiearXiv 2026 · arXiv

TL;DR

AnomalyJev is a lightweight framework that evaluates image anomaly detection explanations by extracting verifiable claims (like defect type and location) and checking them against ground truth data. By ditching expensive, slow, and often inconsistent autoregressive LLM judges in favor of a deterministic verification process, it achieves human-level agreement at 1/150th the cost and 40x the speed.

What Problem Does This Paper Address?

Multimodal Large Language Models (MLLMs) are increasingly used for industrial anomaly detection, providing natural language explanations for why an image is flagged as "anomalous." However, we lack a reliable way to evaluate these explanations.

Standard text metrics like BLEU or ROUGE focus on word overlap rather than factual correctness. On the other hand, using "LLM-as-a-judge" (prompting a powerful model like GPT-4o to score an explanation) is problematic: it’s computationally expensive, slow due to token generation, and often biased toward linguistic fluency rather than factual accuracy. This paper addresses the need for an evaluation method that is faithful (factually correct), complete (covers all relevant details), and scalable.

Methods

The authors propose a two-stage "extract-then-verify" pipeline called AnomalyJev.

  1. Claim Extraction: Instead of asking a model to write a review, AnomalyJev uses a small, fine-tuned LLM (Qwen-0.8B) to extract structured claims from the explanation text. It identifies the predicted verdict (normal/anomalous), the defect type, and the location (mapped to a 3x3 grid). Crucially, it uses pointer heads to select options from a closed set rather than generating new text, making it deterministic and extremely fast.
  2. Claim Verification: These extracted claims are then automatically compared against the actual ground truth (GT) annotations for the image.
Figure 1: LLM/MLLM-as-a-judge vs. claim-based veri- fication. (a) A judge takes the ground truth, evaluation rubric, and explanation as input and autoregressively generates a score. (b) AnomalyJev predicts one option per ground-truth attribute directly from contextualised representations, then verifies the resulting claims to pro- duce deterministic scores, without token generation.

The framework calculates two scores:

  • Faithfulness: Penalizes incorrect claims.
  • Completeness: Rewards the coverage of all ground-truth attributes.

Datasets

The models were trained and tested on several key anomaly detection benchmarks:

  • MVTec-AD & MVTec LOCO: Used for training the extraction module and in-distribution testing.
  • VisA: Used to test cross-dataset generalization (unseen objects and defect types).
  • Human-Annotated Benchmark: The authors created a new evaluation set of 640 explanations generated by 11 different MLLMs, accompanied by over 1,600 expert ratings to serve as a gold standard for validating the evaluator itself.

Key Insights and Ideas

  • Ground Truth is the Anchor: By anchoring evaluation in the available ground truth rather than a reference text, the system avoids the "fluency bias" where judges reward well-written but factually wrong explanations.
  • Generation-Free Efficiency: Moving from autoregressive generation to a pointer-based selection mechanism is the key to its massive speedup. The model doesn't "think" or "write"; it just points to the most likely attributes mentioned in the text.
  • Faithfulness vs. Completeness: The paper highlights that these are distinct dimensions. A model can be 100% faithful by saying very little (e.g., just "it's broken"), but it would have low completeness. Conversely, a detailed explanation might be complete but full of "hallucinated" (unfaithful) details.
  • Small Models Can Judge: A tiny 0.8B parameter model, when properly framed as a claim extractor, can match or beat 30B+ parameter models acting as holistic judges.

Key Results

AnomalyJev demonstrates impressive performance across the board:

  • Human Agreement: It achieves a Spearman correlation ($\rho$) of 0.940 for faithfulness, outperforming most open-source MLLM judges and approaching flagship models like Gemini 3.1 Pro (0.967).
  • Efficiency: It processes an explanation in just 0.11 seconds on an A100 GPU, making it 40x faster than flagship judges.
  • Cost: At approximately $0.06 per 1,000 explanations, it is 150x cheaper than using proprietary APIs like Gemini Pro.
  • Generalization: It maintains high accuracy even when evaluating explanations from MLLMs it wasn't trained on and when applied to entirely new datasets like VisA.
Figure 3: Agreement with human experts (Spearman ρ, faithfulness) against model size and runtime.

Limitations and Future Work

  • Grid Granularity: The current 3x3 grid for location verification is relatively coarse. It cannot capture fine-grained spatial descriptions or relative positions (e.g., "to the left of the screw").
  • Ground Truth Dependency: The system can only verify what is in the ground truth. If an explanation mentions a detail not covered by the annotations (e.g., the number of anomalies), it remains unverified.
  • Domain Specificity: While it generalizes well to other industrial datasets, applying it to vastly different domains (like medical imaging) would likely require new training data for the extraction module.
  • Future Directions: The authors suggest that future work could involve finer spatial reasoning and expanding the set of verifiable attributes beyond just type and location.

Summary updated

Abstract

Recent multimodal LLM (MLLM)-based anomaly detection (AD) methods can generate textual explanations that help interpret their predictions. However, reliably and efficiently evaluating these explanations remains challenging. While LLM judges can assess explanations through structured rubrics, they often require substantial token generation, making evaluation costly and difficult to scale. We instead propose to anchor evaluation in the available ground truth by extracting verifiable claims from each explanation and verifying them against the corresponding annotations. We instantiate this idea as AnomalyJev, a lightweight framework that extracts verifiable claims from contextualised representations using ground-truth-defined queries and pointer based heads, without token generation. The claims are then verified against the ground truth to score faithfulness and completeness. For evaluation, we construct a human-annotated set of 640 explanations with over 1,600 expert ratings across in-domain and cross-dataset settings. Despite its compact design, AnomalyJev achieves stronger agreement with human annotators than most open MLLM judges and approaches that of flagship MLLM judges, at a fraction of their runtime and resources.
BibTeX
@misc{wang2026anomalyjev,
  author = {Qizhou Wang and Jiawen Zhu and Guansong Pang and Christopher Leckie},
  title = {{AnomalyJev: Towards Evaluating Image Anomaly Explanations via Claim-Level Verification}},
  year = {2026},
  eprint = {2610.27565},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.27565},
}
Figure