Same Outputs, Different Scores: Auditing the Unchecked Graders of Multimodal LLM Benchmarks

Ahmad Jarjis Hasan, Jonathan LwowskiarXiv 2026 · arXiv

TL;DR

This paper reveals that the accuracy scores of multimodal LLMs (MLLMs) are heavily influenced by hidden "scoring rules" used to parse free-form responses into multiple-choice verdicts. By auditing eight major benchmarks, the authors found that these unexamined rules can swing a model's accuracy by up to 22 points on the exact same outputs, effectively conflating a model's actual capability with the luck of the grader's fallback logic.

What Problem Does This Paper Address?

Most MLLM benchmarks are multiple-choice, but models—especially smaller, locally deployable ones—don't always output a clean "(A)" or "(B)". Instead, they often "ramble," providing a sentence like "The object is a nail" or reasoning through a thought process before (or instead of) giving a letter.

When the parser fails to find a clear option letter, the benchmark's "grader" must decide what to do. This "parse-failure fallback" is usually a few lines of code buried in a scoring script, chosen without justification and rarely reported. As shown in Figure 1, the same rambling response can be marked "Correct," "Wrong," or "None" depending on which benchmark's grader is used.

Figure 1. The grade comes from the grader, not the model. A single rambling response receives contradictory verdicts un- der four scoring rules shipped by published benchmarks; none matches what a blinded human says the model answered. On a compliant response the rules agree, which is why the artifact is in- visible on leaderboards of strong, format-following models.

This creates a massive "artifact" in reported leaderboards. Because frontier models (like GPT-4o) are highly instruction-tuned to follow formats, the artifact is invisible at the top of the leaderboard. However, for the 7B–13B class models that researchers and engineers actually deploy locally, these grading choices become the primary driver of the final score.

Methods

The authors performed a "code audit" of the official scorers for eight peer-reviewed benchmarks. They reproduced these scorers and tested them on five open Vision-Language Models (VLMs) that fit on a single 16GB GPU (e.g., Llama-3.2-11B, Pixtral-12B).

To establish a "ground truth" of what the models actually meant, they introduced a blinded human-plus-LLM-judge adjudication protocol. A human and a frontier LLM (Claude 3.5 Opus) independently labeled model responses as a specific option, "NONE" (no commitment), or "MULTIPLE" (contradictory). This protocol achieved high agreement (Cohen’s $\kappa = 0.956$), allowing the authors to measure how often automated graders over-credit or under-credit models compared to a human reading of the same text.

Datasets

The audit covered a wide range of popular MLLM benchmarks:

  • General/Reasoning: MMMU, ScienceQA, MathVista, MathVision.
  • Industrial/Medical: MMAD (Industrial Anomaly Detection), OmniMedVQA, PMC-VQA.
  • Hallucination: POPE.

The authors specifically used MMAD and PMC-VQA for their human-anchored adjudication because their formats (binary and 4-option) make human agreement highly reliable.

Key Insights and Ideas

  • The "Snap" Artifact: Many benchmarks use a "no-threshold snap" (e.g., get_close_matches) that forces any string to the nearest option. On binary tasks, this heavily inflates scores by rewarding models that ramble without ever committing to an answer.
  • Bidirectional Errors: The same rule can help one model while hurting another. In Figure 3, the authors show that a snap rule adds 22 points to one model on MMAD but removes 19 points from another on PMC-VQA by overriding correct wordy answers with incorrect "nearest" letters.
  • Likelihood is not a Silver Bullet: Scoring based on the model's logits (likelihood) is often seen as more "principled," but the authors found it is frequently biased by token-priors (e.g., a model preferring the token "A" regardless of the content).
  • Reasoning Models aren't Immune: Even for models with 0% non-compliance (they always output a letter), the score can swing by 9.5 points based on whether the parser keeps the first or last regex match in a long chain-of-thought trace.
  • The "Invisible" Problem: On leaderboards dominated by format-compliant frontier models, these grading artifacts are hidden. They only emerge when evaluating the "messier" open-source models used in specialized, air-gapped, or privacy-sensitive applications.

Key Results

The most striking result is the magnitude of the swing: swapping the parse-failure rule alone moves the accuracy score by up to 22 percentage points on identical model outputs (Figure 2).

Figure 2. The scoring rule alone moves the score by up to 22 points. MMAD accuracy on identical stored model outputs under three parse-failure scoring rules. The three format-compliant models score the same under every rule (gap = 0); on the two models that ramble, the no-threshold closest-option snap inflates accuracy by 21–22 points over abstention, with the likelihood rule different again. Exact on all 1,670 items, deterministic given the outputs.

On the MMAD benchmark, the "no-threshold snap" (the official rule) scored Llama-3.2-11B at 0.741, while an "abstention" rule (marking non-compliant answers wrong) scored the exact same outputs at 0.522.

Furthermore, the authors found that a single character of unreported regex in the ScienceQA benchmark was worth 29.5 points for one model, illustrating how fragile these metrics are to implementation details.

Limitations and Future Work

The authors acknowledge several boundaries:

  • Model Size: The study focused on 7B–12B models; while the artifact is smaller on frontier models, it doesn't disappear, it just "migrates" from the fallback logic to the parser logic.
  • Task Complexity: The human-anchored ground truth is strongest on binary and 4-option tasks. Scaling this to more complex open-ended tasks remains a challenge.
  • Ranking: While the rules change absolute scores significantly, they did not find a "robust leaderboard reordering" for the very top-ranked models, though the gaps between mid-tier models were often entirely erased or reversed.

The paper concludes with a call for better reporting: benchmarks should report a non-compliance rate and provide accuracy scores under at least two different scoring rules to show the sensitivity of the results.

Summary updated

Abstract

Every multimodal-LLM multiple-choice benchmark ends with a scoring rule that converts a model's response, often free-form text rather than a clean option letter, into a right/wrong verdict, buried in its code. Unlike the dataset, this rule is chosen without justification and left unexamined, yet it sets the published number. We reproduce the official scorers of eight peer-reviewed benchmarks on five open vision-language models (VLMs) that fit a single 16 GB GPU, and introduce a blinded human-plus-LLM-judge adjudication (Cohen's kappa=0.956) that recovers what the model actually answered. The scoring rule alone moves accuracy by up to 22 points on identical outputs, yet exactly 0 wherever a model emits clean option letters, so it hides on strong-model leaderboards. It is the fallback and not the rambling that does the damage: the benchmark with the highest non-compliance we measure has an artifact of exactly zero, because its scorer abstains instead of guessing; a reasoning model with non-compliance 0.000 still swings 9.5 points with which regex match of its trace the parser keeps. No standard rule is reliable: snapping a non-answer to its closest option credits answers never given yet buries correct ones, abstention discards answers stated in words, and likelihood scoring follows answer-token priors, so a single grade conflates capability with luck rather than measuring either. We release all code, labels, and the protocol, and argue benchmarks should report a non-compliance rate and at least two rules, not one unexamined grade.
BibTeX
@misc{hasan2026same,
  author = {Ahmad Jarjis Hasan and Jonathan Lwowski},
  title = {{Same Outputs, Different Scores: Auditing the Unchecked Graders of Multimodal LLM Benchmarks}},
  year = {2026},
  eprint = {2611.03377},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.03377},
}
Figure