Do Hallucination Detectors Catch Errors or Just Absurdity? A Humicroedit Stress Test

Detectors of large language model (LLM) hallucination, meaning fluent statements that are not supported by the facts, are almost always tested on errors that some model happened to produce. Nobody controls how surprising or implausible those errors are. We propose a cleaner probe: the Humicroedit headlines. Each one is a real news headline in which a person replaced exactly one word to make it funny. That gives thousands of minimal pairs: a true sentence and a false sentence of the same length, differing in one slot. Human raters scored each edit for funniness, and funniness in this data tracks how badly the new word clashes with its context. Our hypothesis is that current detectors, including LLMs prompted to act as judges, mostly respond to clash. If so, they will reliably flag absurd edits like "Senate votes to repeal pizza". They will miss mundane, realistic swaps like a wrong country or a wrong official, and those mundane swaps are what real hallucinations usually look like.

The second half of the claim concerns ensembles. Combining many detectors is the standard way to cover any one detector's blind spots, and it works when their errors are uncorrelated. If every detector shares the same reliance on clash, combining them will sharpen performance on absurd edits, where it is not needed, and leave the realistic tail as bad as before. Humicroedit lets us measure exactly that: how correlated the detectors' errors are, broken down by how realistic the edit is. In two sentences: one-word humorous edits give a length-controlled, human-graded continuum from absurd to realistic factual deviation. We use it to test whether hallucination detectors, and ensembles of them, are measuring plausibility rather than truth.

Core Claim

We claim three things, and each could turn out false. First, the probability that a detector flags an edited headline rises with human-rated incongruity. It is lowest for edits people judge to be realistic headlines. Second, this dependence survives after we control for a language model's word surprisal, which measures how unexpected the word is. If it does, detectors encode something closer to human-perceived absurdity than raw token improbability. Third, in the realistic bin, errors across judges from different model families are highly correlated. As a result, averaging and stacking help mainly where single detectors already succeed. The constructive question is whether a detector trained on alignment with a source text rather than on plausibility breaks that correlation. That detector is AlignScore, a model trained on 4.7M examples to label a claim as aligned with, neutral to, or contradicting a source.

Why this intersection, and what is already done

Humicroedit comes from SemEval-2020 Task 7, which built 15,095 one-word edits of 5,000 headlines, each rated 0–3 by five annotators. Its baseline analysis found that larger GloVe distance between the original and replacement word predicts higher funniness. That finding is what makes funniness usable as a graded incongruity signal. The same paper reports a U-shaped error curve: models do worst at both ends, including at 0. This warns us that "unfunny" mixes two different things, edits too mundane to be funny and edits too nonsensical to be funny. For that reason, funniness alone cannot serve as our realism axis, and we collect a separate realism rating, described below.

Work on hallucination evaluation motivates the question but has not asked it. HaluEval has ChatGPT generate hallucinations and keep only the plausible ones. It then shows that ChatGPT judges detect them poorly: 62.59% accuracy in QA and 58.53% in summarization. Because HaluEval filters for plausibility, it never varies plausibility, so it cannot show that plausibility drives detection. The Illusion of Progress shows that ROUGE-based evaluation inflates detector scores. Under an LLM judge, Perplexity loses up to 45.9% AUROC and Eigenscore loses 19.0–30.4%. It also shows that response length alone rivals complex detectors. That length finding constrains our design: we need a setting where length carries no signal, and Humicroedit holds length fixed by construction. Getting Serious about Humor shows that LLMs can "unfun" satire back into realistic headlines. It also contributes human realism ratings: 49% of GPT-4 unfuns were judged realistic, against 33% of human edits. We reuse their realism protocol and their minimal pairs as a second test set.

On the ensemble side, Hitachi's Stacking at Scale won SemEval-2020 Task 7 by stacking hundreds of fine-tuned language models under a Ridge regressor. RedwoodNLP found that lightweight models beat transformer ensembles on subjective targets. Auto-Prompt Ensemble for LLM Judge combines prompt variants to reduce single-prompt bias. All three ensemble for accuracy on one target. None asks whether ensemble members fail together on a specific, controllable slice of inputs. We borrow Hitachi's out-of-fold stacking as a method and use it to test that question. Nothing in the file runs any hallucination detector on micro-edited humor, and we found no reported result on Lacuna for hallucination detection on Humicroedit.

Research Program

The probe set. From Humicroedit we draw a stratified sample of about 2,000 edits spread across the funniness range. We add a matched set of Unfun minimal pairs. For each edit, five crowd raters answer the realism question from Horvitz et al.: "could this plausibly be a real headline?" This yields two human axes per edit, funniness and realism. We also compute three automatic incongruity measures for the replaced slot: GloVe cosine distance, masked-language-model surprisal, and surprisal under the detector model itself. As a control, we build about 1,000 synthetic entity swaps of the kind that cause real hallucinations, such as same-type named entities or wrong numbers. These are rated for realism the same way, so that our conclusions do not depend on what humans choose to edit when they are trying to be funny.

Two detection settings. In the reference-based setting, the detector sees the original headline as the source and must judge whether the edited headline is consistent with it. Every edit is a contradiction, so any miss is a pure failure. In the closed-book setting, the detector sees only one headline and must judge whether it is factually accurate. This mirrors hallucination detection without retrieval. Headlines date from 2017–2019 news, so most current models have had a chance to see the true versions. The paired design gives a clean metric: for each pair, does the detector score the edited version as less factual than the original?

Detectors. We test three groups. The first is prompted LLM judges from at least three model families, each with a step-by-step rubric prompt and a prompt-ensemble variant. The second is uncertainty-based detectors: perplexity, sampling-based self-consistency in the style of Agrawal et al., and semantic entropy. The third is AlignScore, run through yuh-zha/AlignScore. In the reference-based setting, AlignScore was trained to detect contradiction with a source, not oddness.

Analysis. The central quantity is how detection probability depends on realism. We fit a logistic model per detector:

$$ \Pr(\text{flag}_i) = \sigma\big(\beta_0 + \beta_R R_i + \beta_S S_i\big), $$

Here $R_i$ is the mean human realism rating of edit $i$, $S_i$ is its standardized surprisal under a fixed reference model, and $\sigma$ is the logistic function. Claim one predicts $\beta_R < 0$. Claim two predicts that $\beta_R$ stays clearly negative with $S_i$ in the model. That would mean detectors track human-perceived plausibility beyond token surprise. For the ensemble analysis, we compute the pairwise correlation of detector errors separately within each realism tercile. We then compare the gain from ensembling over the best single member in each tercile.

Baselines

The do-nothing reference is chance on the paired task, 50%. The surface-incongruity baseline flags an edit whenever its GloVe distance or surprisal exceeds a threshold tuned on a held-out split. If this baseline matches the LLM judges bin by bin, then those judges add nothing beyond token oddness, and that is itself a result. The reported point of comparison for LLM judges is HaluEval. There, a single ChatGPT judge reaches 62.59% accuracy on QA and 58.53% on summarization (HaluEval). Those numbers come from model-written plausible errors, so we expect our realistic tercile to land in that range and our absurd tercile well above it.

The strongest existing method for the reference-based setting is AlignScore-large. It reports an average AUC-ROC of 88.6 on SummaC and 87.4 on TRUE (AlignScore). Its pairwise accuracy on our realistic tercile is the number to beat. The strongest ensemble baseline is Hitachi-style out-of-fold stacking with a Ridge meta-learner over all detector scores (Hitachi), trained with 5-fold cross-validation on the probe set. We also run simple score averaging and the prompt ensemble of Auto-Prompt Ensemble.

All methods see the same pairs and the same folds. Every LLM judge gets the same budget of prompt variants and the same number of samples. Thresholds are tuned only on training folds. The key ablation removes AlignScore from the stack and compares the realistic-tercile gain with and without it. This isolates whether decorrelation, rather than adding more members, drives any improvement. A method counts as "beating" the baselines only if it improves realistic-tercile pairwise accuracy by more than the bootstrap 95% interval while staying close to its accuracy in the absurd tercile.

Why not just…?

Why not just use surprisal or GloVe distance and skip the LLMs? That is our baseline, and it is also the variable we suspect detectors secretly rely on. The Illusion of Progress already shows perplexity is brittle as a hallucination signal. Surprisal by construction cannot flag a realistic swap, which is low-surprisal by definition. The question is whether LLM judges do any better than that. The regression with $S_i$ as a control answers it directly.

Why not just generate plausible entity swaps synthetically, as HaluEval does? This is a real threat. If our only finding were "plausible errors are hard," synthetic swaps would suffice, so we include them as a control set. Humicroedit adds two things synthetic generation cannot easily provide. It gives a human-graded continuum running from absurd to realistic, which lets us measure a slope rather than one point. And its edits were written by people, so the difficulty does not come from an LLM picking errors that fool LLMs. If the synthetic control and Humicroedit produce the same slope, we will report that the humor data adds coverage of the continuum but nothing qualitatively new.

Why not just use one strong frontier judge? HaluEval's 58.53–62.59% numbers suggest that single judges have blind spots. Whether a frontier judge's blind spot is the realistic tercile is exactly what our per-tercile analysis measures. If one judge is uniformly strong across terciles, the ensemble claim fails, and we will say so.

Isn't this just length bias again? No. Length is identical within each pair, so the length heuristics that The Illusion of Progress found competitive carry no signal here.

Decisive Checks

The proposal stands or falls on three measurements. First, a flat realism slope ($\beta_R \approx 0$) for LLM judges in the reference-based setting would refute the central claim. Second, $\beta_R$ shrinking to zero once surprisal is controlled would reduce the claim to "detectors track token surprise," which is a weaker finding. Third, if ensemble gains in the realistic tercile match those in the absurd tercile, the correlated-failure claim is false. If all three come out as predicted, Humicroedit becomes a cheap standing benchmark for a specific question: does a hallucination detector catch falsehood, or only weirdness? It would also give direct evidence for which kinds of ensemble members actually cover the gap.