Mining Compact Failure Rules for Multimodal Judges

Large AI models are increasingly used as graders. They score other models' answers about images, and those scores steer training and benchmark leaderboards. We already know such "multimodal judges" have blind spots. They trust fluent text over what the image shows. They do worse looking at raw pixels than at a written description of the same picture. And, in plain recognition tasks, they fall back on the ethnicity of the person in a photo rather than the object that person is holding.

What we lack is a way to say exactly when a judge goes wrong. That means a short, readable list of conditions such as "image shows a low-resource country's dish, the person's appearance does not match its origin, and the answer is long and confident." Each such condition should be checked to see whether it actually causes the failure. This proposal builds that tool. It adapts compression-based rule mining, a data-mining technique that finds the few rules that best summarize where errors occur, to judge verdicts. It then confirms each mined rule with targeted image and text edits.

The two-sentence version is this. We treat every disagreement between a judge and the truth as a data point. We search for the smallest set of feature combinations that explains those disagreements, where the features span demographic content of the image, style of the graded answer, and whether the judge saw pixels or a caption. Because naive rule search on rare subgroups produces thousands of spurious rules, we keep only rules that survive a counterfactual edit test. Those rules then become a targeted training set for repairing the judge.

Core Claim

Systematic failures of multimodal judges on demographically and culturally varied content are combinations of a few factors, not single factors. These factors span three sources: what is in the image, how the graded answer is written, and which input pathway the judge uses. A compression-based miner restricted by counterfactual verification can recover these combinations from a few thousand judged items. It will recover them more faithfully than embedding clustering, factorial sweeps, or asking an LLM to guess failure modes. Training data aimed at the recovered rules should repair the judge more efficiently than equal-size data built from fixed templates.

The claim rests on three findings that do not yet meet. Label-Descriptive Patterns and Their Application to Characterizing Classification Errors introduced PREMISE. PREMISE finds a compact set of AND/XOR rules over input tokens that describe where a classifier errs, chosen by Minimum Description Length (MDL), meaning the rule set that most shortens an encoding of the error labels. It found text shortcuts in visual question answering models. It worked on discriminative models, used only input tokens, and never touched generative judges or demographic attributes. We reuse its method.

MLLM-as-a-Judge showed that judges often score better from a text description than from the image. It also showed that step-by-step reasoning can make judges invent visual details. Mitigating Perceptual Judgment Bias showed that fluent answers can talk a judge out of what it perceives. These two papers motivate the work: the gap between pixel input and text input is a measurable signal of judge failure. Neither paper asks which conditions trigger it, and both stay within STEM, chart, and general-domain content.

When Tom Eats Kimchi built MIXCUBE. It swaps the ethnic appearance of people in images while keeping the cultural artifact fixed, and finds recognition collapses when appearance and origin mismatch. That work supplies our counterfactual machinery and defines the application regime. It tested recognition, not judging, and it used a pre-chosen four-category factorial design rather than discovering combinations.

Research Program

Stage 1: build judged items with known ground truth. Build a judging set on MIXCUBE images, whose artifact origin is known by construction. For each image and question, generate graded answers along three axes:

  • correct versus shortcut-wrong (for example, naming the culture implied by the person's appearance);
  • terse versus verbose;
  • plain versus pseudo-authoritative.

The style axes follow the perturbation types catalogued by CALM in Justice or Prejudice?. CALM tested them only on text. Because truth is known, every judge verdict can be labeled right or wrong without human raters, in the spirit of how PPJD was built. We add the MLLM-as-a-Judge HARD split and PPJD to test whether rules generalize beyond cultural content.

Stage 2: record each judge three ways. For every item we record the judge's verdict from raw pixels and from a strong captioner's description, the "vision expert" setting of MLLM-as-a-Judge. We also record the verdict after an appearance swap. These yield three binary labels per item:

  • the judge was wrong;
  • the pixel and description verdicts disagree;
  • the verdict flipped under the swap.

Stage 3: describe each item with discrete features. Features include the depicted appearance category, the artifact's country and resource level, domain, objects and attributes tagged by a detector, answer length bin, citation presence, sentiment, rubric type, and judging protocol (score, pairwise, or ranking).

Stage 4: mine rules. Run PREMISE-style MDL mining on each label. The miner selects the rule set $P$ that minimizes

$$L(P) + L(Y \mid P),$$

where $L(P)$ is the bits needed to write down the rules and $L(Y \mid P)$ is the bits needed to encode the error labels $Y$ given those rules. Short rule sets that explain many errors therefore win.

Stage 5: verify rules by counterfactual edits. This is the new ingredient. Its purpose is to make low support usable. The interesting failures concern minority cultures and appearances that make up a small share of items. As the research file notes, lowering the support threshold there makes the number of candidate rules explode with chance correlations. Rather than relying only on Fisher's exact test as PREMISE does, we take each candidate rule and edit fresh items:

  • re-inpaint appearance with the MIXCUBE pipeline;
  • rewrite answer style;
  • swap the artifact.

The edits switch the rule's conditions on and off while holding everything else fixed. We keep a rule only if switching it on raises the failure rate by a margin with a confidence interval excluding zero. Surviving rules are then fed back as constraints to the miner, which re-runs. This loop turns a correlational summary into a short list of causally checked triggers.

Stage 6: repair the judge. Use the verified rules to generate a targeted counterfactual training set. Train an open judge (Qwen2.5-VL-7B) with the ranking-reward GRPO recipe of Perception-Judge. GRPO, group relative policy optimization, is a reinforcement learning method that scores groups of sampled outputs against each other. The training pushes the judge to rank correct answers above shortcut answers, regardless of style or appearance.

Baselines

Naive reference. The do-nothing reference is the off-the-shelf judge, scored on the same items. For general judging, MLLM-as-a-Judge reports GPT-4V at Pearson 0.490 for scoring and about 79% pairwise agreement with humans. On PPJD, Mitigating Perceptual Judgment Bias reports 69.5% accuracy for base Qwen2.5-VL-7B. For the demographic effect in recognition, MIXCUBE reports a 7.64% average accuracy drop under appearance swap and swings up to 58% for low-resource artifacts with GPT-4o. We found no reported result on Lacuna for appearance-swap flip rates of judges, so we establish that number first.

Discovery baselines. For the discovery question, the strongest existing slicing method in this space is the cluster-based partition of Calibrating MLLM-as-a-judge via Multimodal Bayesian Prompt Ensembles. It groups images by spherical k-means over CLIP embeddings (saturating at 16–32 clusters) and reports a 36% expected calibration error reduction versus an averaged-prompt baseline on HPSv2. We run it on our judged items with the same cluster budget and treat high-error clusters as its "discovered slices."

The second discovery baseline is unmodified PREMISE on the same features, without the verification loop. Its reported synthetic rule-recovery F1 is above 0.90 on classifier errors. This doubles as the ablation that isolates our key ingredient. We also include RIPPER, which PREMISE outperformed.

Fair comparison and planted-rule test. All discovery methods get the same items, the same features (clusters additionally see embeddings), and the same number of judge calls. That includes the calls our verification loop spends on edits, which are given to baselines as extra labeled data. Beating them means higher F1 on a planted-rule test and higher precision of rules under fresh counterfactual edits. In the planted-rule test, we inject known combinatorial score corruptions into a judge's outputs at support levels from 0.5% to 5% and measure recovery.

Repair baselines. For repair, the strongest competitor is Perception-Judge's template-built PPJD training, which reaches 85.7% on PPJD. We compare at equal training-set size and equal GRPO compute. We also compare against a random-counterfactual set of equal size. Beating them means a lower held-out appearance-swap flip rate and a smaller pixel-versus-description gap, without falling below 85.7% on PPJD.

Why not just…?

Why not just run a full factorial sweep, as MIXCUBE does, and report every cell? This is the main threat. With four appearance categories and a handful of style variables, a sweep is cheap and complete, and mining adds nothing. The case for mining begins once features include detector-tagged objects, settings, rubric types, and protocols. At that point cells number in the thousands, most are empty, and per-cell estimates are noise. We settle the question empirically. On the low-dimensional subset, we compute both and expect agreement. On the full feature set, we check whether mined rules predict flips on held-out edits better than the top-k factorial cells ranked by observed error. If they do not, the honest conclusion is that a sweep suffices, and we will report that.

Why not just cluster judge inputs by embedding and inspect bad clusters? The prompt-ensemble paper shows clustering helps calibration, but its clusters are regions of global image space. Our hypothesized triggers mix image content with answer style and input pathway, and an image embedding cannot see answer style at all. The planted-rule test directly measures whether clusters recover combinatorial triggers. We expect them to do poorly whenever a trigger involves a style feature.

Why not just show an LLM a batch of failures and ask it to name the patterns? This is cheap and may produce plausible hypotheses. However, the research file's own evidence suggests those hypotheses would carry the same priors we are auditing. Judges anchor on fluent text (Mitigating Perceptual Judgment Bias), and recognizers default to demographic shortcuts (When Tom Eats Kimchi). We include this as a baseline anyway, verifying its proposed rules with the same counterfactual edits. If its verified-rule precision matches ours, the contribution narrows to the verification loop. The ablation with unmodified PREMISE tells us whether that loop is the real source of value.

Why not just train a distributional judge that absorbs disagreement? Distributional LLM-as-a-Judge learns the spread of human ratings (SNLI KL 0.23 versus 0.60 for single-point training). It treats disagreement as an unexplained probability vector, however, and works only on text. Our problem is a judge that is confidently wrong in specific, nameable conditions, which a better-calibrated spread would not reveal.

Proposed Experiments

Experiment 1: does the phenomenon exist in judging? We measure appearance-swap flip rates and pixel-versus-description disagreement for GPT-4o, InternVL2.5-26B, and Qwen2.5-VL-7B judging MIXCUBE-derived items. Results are split by country resource level. We expect flips to concentrate on low-resource artifacts, mirroring MIXCUBE's recognition result. We also expect flips to be amplified when the shortcut-wrong answer is verbose or authoritative, which would be the interaction no existing paper has measured.

Experiment 2: does discovery work? This is the planted-rule recovery test plus the fresh-edit precision test, comparing our method against unmodified PREMISE, RIPPER, k-means slices, and LLM-proposed rules. We report rule-recovery F1 against PREMISE's reported >0.90 synthetic figure. We also report how the number of reported rules and their precision change as minimum support drops from 5% to 0.5%. The decisive check is whether verification keeps precision flat while unverified mining's rule count explodes and precision falls.

Experiment 3: do mined rules transfer? We test whether rules mined on cultural content predict failures on MLLM-as-a-Judge HARD and PPJD. This includes asking whether the "verbose plus contradicts image" pattern recurs outside demographic content.

Experiment 4: does targeted repair beat generic repair? We compare rule-targeted GRPO training against PPJD-template and random-counterfactual training at matched size. The outcomes are held-out swap flip rate, pixel-versus-description gap, PPJD accuracy (with 85.7% as the bar), and Pearson agreement on MLLM-as-a-Judge.

The proposal fails cleanly in two cases. One is if targeted data does no better than random counterfactuals. The other is if the factorial sweep matches mined rules on held-out prediction. Either outcome would tell the field that the cheap alternative is enough.