Checking Anomaly Alarms with an LLM Judge That Reads the Detector's Own Evidence

Fast anomaly detectors, such as Isolation Forests (tree ensembles that flag points that are easy to isolate with random splits) or autoencoders, give each data point a score and nothing else. Operators then face two questions the score cannot answer: is this alarm real, and is the stated reason for it believable? This proposal tests a specific division of labor. The detector and a standard attribution method do the numeric work, reporting which features pushed the score up and where their values sit relative to training data. A mid-sized language model then acts as a judge. It reads that evidence packet, together with the real-world names of the features, and decides whether the alarm describes a plausible anomaly or a spurious artifact.

The bet rests on a result from TSQueryBench, which found that language models are much better at checking candidate explanations of time-series anomalies than at writing them. Qwen-3 8B reached 0.84 accuracy when ranking candidate explanations but only 0.38 when generating them. If that asymmetry carries over to tabular alarms, the model should not be asked to find anomalies. It should be asked to verify evidence that a cheap detector has already produced. The main risk is also what makes the question clean. If the judge does just as well when feature names are replaced with "f1…f36", it is only doing arithmetic that a small classifier could do, and the idea fails.

Core Claim

An LLM judge that receives attribution-grounded evidence for each alarm, together with feature semantics, will rerank a detector's alarms better than two alternatives. The first is the detector alone. The second is a small classifier trained on the same handful of labeled anomalies. The gain should come specifically from the semantics: it should shrink toward zero on datasets whose features have no meaningful names. As a secondary claim, when the judge is shown two competing attributions for the same alarm, it will more often pick the one that matches the true cause of an injected anomaly.

What is new is the object being judged. TSQueryBench judged free-text explanations of synthetic univariate series. Grounded TSAD trained a 3B multimodal model to detect and explain its own univariate anomalies. LLMAD used GPT-4 as the detector itself. None of them takes the output of an external black-box detector and its attribution method as the thing to be verified. None of them works on multivariate tabular data, where dependency anomalies occur: points whose individual feature values look normal but whose combination does not.

Research Program

For each alarm raised by a base detector, we assemble an evidence packet with four parts:

  1. the top-k attributed features from AcME-AD, a fast perturbation method that probes the detector using training-set quantiles, or from DIFFI, which reads importances off Isolation Forest split depths;
  2. each feature's value and its training quantile;
  3. AcME-AD's "distance to change", meaning how far a feature must move for the alarm to flip;
  4. the feature names and a one-line dataset description.

Every numeric claim in the packet uses the machine-checkable citation format of Grounded TSAD, adapted to tabular data as (feature:value:quantile). The judge must cite from the packet, and a script rejects any verdict whose citations do not match the packet. This is our answer to the tension between strict grounding and open-ended rubrics: numbers are checked mechanically, and only the plausibility reasoning is left free.

The judge follows a short rubric, in the style of TSQueryBench's dimensions, adapted to alarms. It asks whether the attributed deviation is real, whether it is physically or semantically coherent, and whether the combination of features suggests a known failure or an artifact. The judge gives brief reasoning and then a verdict on a 1–5 scale. We use Qwen-3 8B and Gemma-4 31B, the strongest judges in TSQueryBench, plus one frontier model as a ceiling. If the gain holds, we distill the judge into a cheaper model, following the budget motivation of LLM-as-a-Judge on a Budget.

Evaluation uses ADBench (ADBench) datasets split into two groups. The first group has semantically named features, such as Glass, Cardio, Thyroid and breast-cancer data. The second group has effectively anonymous features, such as Ionosphere, Musk and Satellite spectral bands. For the attribution task, we reuse ADBench's protocol for injecting Local, Global, Dependency and Clustered anomalies into real normal data, so that the true causal features are known. AD-LLM gives prior reason to expect semantics to matter: its text detectors exceeded 0.95 AUROC only when told the anomaly category name. The Glass result in DIFFI, where Aluminum and Barium are the true headlamp markers and SHAP ranked Magnesium second, gives a concrete case where domain knowledge separates good attributions from bad ones.

Two known judge failures shape the design. The Silent Judge shows that judges favor length and polish. We therefore fix the packet format and randomize feature order. Investigating Non-Transitivity in LLM-as-a-Judge shows that pairwise preferences can form cycles. We therefore report cycle rates in the three-way attribution comparison (SHAP vs DIFFI vs AcME-AD) rather than assuming a clean ranking exists.

Baselines

The naive reference is the base detector's own score ordering (Isolation Forest, and one deep detector) with no second stage. Any reranker must improve precision on the top alarms relative to this.

The strongest cheap competitor is a semi-supervised second stage: a gradient-boosted classifier given the detector score, the same attribution vector, and the quantile features. It is trained on the same labeled anomalies the judge sees as few-shot examples, fixed at 1% of anomalies per dataset. ADBench reports that semi-supervised methods reach a median AUCROC of 75.56% with 1% labeled anomalies, against 60.84% for fully supervised methods (ADBench). This is the reported figure closest to our setting, although it is for detection, not alarm reranking. No reported result was found on Lacuna for LLM-based reranking of tabular anomaly alarms. For fairness, the classifier's hyperparameters are tuned with the same number of validation evaluations as the judge's prompt variants. Beating it means higher AUROC and precision-at-k over the alarm set on the semantically named datasets, with non-overlapping bootstrap intervals.

The learned-triage competitor is FoMo-X. It attaches small severity and uncertainty heads to a frozen tabular outlier foundation model and reports a 15.15% mistake rate on its "Likely Normal" tier. We treat its severity tier as an alternative reranking signal.

The LLM-as-detector competitor, in the style of LLMAD, gives the same judge model the raw row and training summary statistics but no attribution.

Two ablations isolate the proposal's ingredients. In the first, feature names are anonymized while all other inputs stay identical. In the second, the attribution section is removed.

Why not just…?

Why not just train a small classifier on the attribution vector? This is the real threat. On anonymous-feature datasets we expect it to match or beat the judge, because there the judge has only numbers to work with. The proposal survives only if the judge wins on named datasets and the anonymization ablation removes that win. If the classifier wins everywhere, we report that and drop the claim.

Why not just measure attribution faithfulness numerically, for example with deletion tests? Deletion tests measure whether an attribution matches the model. They cannot tell a faithful explanation of a Clever-Hans detector, one that latches onto dataset artifacts, apart from a faithful explanation of a genuine fault. A Survey on Explainable Anomaly Detection names this as unsolved. The injected-anomaly task directly measures whether the judge adds information beyond faithfulness. We run deletion fidelity alongside it as a competing selector.

Why not just let the LLM detect anomalies directly, as LLMAD does? TSQueryBench found that models asked to find anomalies had high sensitivity but hallucinated extra ones. LLMAD also pays per-point inference cost. The raw-row baseline tests whether this weakness appears on tabular data.

Decisive Experiments

The central measurement is the judge's gain over the semi-supervised classifier, in alarm-reranking AUROC and precision-at-10, split by named versus anonymous datasets. The proposal predicts two things: a positive gain on named datasets, and a gain that shrinks toward zero on anonymous datasets. It shrinks with anonymized names on the same data. A gain that persists after anonymization would mean the judge is not using semantics. A gain absent even on named data would mean the asymmetry does not transfer.

The secondary measurement is accuracy at choosing the attribution that matches the injected features, broken out by anomaly type. Dependency anomalies are the main target, since single-feature quantiles cannot reveal them and TSQueryBench-style univariate judging never tested them. We also report the citation-rejection rate, order-sensitivity under feature permutation, cycle rate in pairwise attribution choices, and wall-clock cost per alarm relative to AcME-AD's roughly 1.5 seconds per explained instance on Satellite.