TL;DR
This paper reveals that the accuracy scores of multimodal LLMs (MLLMs) are heavily influenced by hidden "scoring rules" used to parse free-form responses into multiple-choice verdicts. By auditing eight major benchmarks, the authors found that these unexamined rules can swing a model's accuracy by up to 22 points on the exact same outputs, effectively conflating a model's actual capability with the luck of the grader's fallback logic.
What Problem Does This Paper Address?
Most MLLM benchmarks are multiple-choice, but models—especially smaller, locally deployable ones—don't always output a clean "(A)" or "(B)". Instead, they often "ramble," providing a sentence like "The object is a nail" or reasoning through a thought process before (or instead of) giving a letter.
When the parser fails to find a clear option letter, the benchmark's "grader" must decide what to do. This "parse-failure fallback" is usually a few lines of code buried in a scoring script, chosen without justification and rarely reported. As shown in Figure 1, the same rambling response can be marked "Correct," "Wrong," or "None" depending on which benchmark's grader is used.
This creates a massive "artifact" in reported leaderboards. Because frontier models (like GPT-4o) are highly instruction-tuned to follow formats, the artifact is invisible at the top of the leaderboard. However, for the 7B–13B class models that researchers and engineers actually deploy locally, these grading choices become the primary driver of the final score.
Methods
The authors performed a "code audit" of the official scorers for eight peer-reviewed benchmarks. They reproduced these scorers and tested them on five open Vision-Language Models (VLMs) that fit on a single 16GB GPU (e.g., Llama-3.2-11B, Pixtral-12B).
To establish a "ground truth" of what the models actually meant, they introduced a blinded human-plus-LLM-judge adjudication protocol. A human and a frontier LLM (Claude 3.5 Opus) independently labeled model responses as a specific option, "NONE" (no commitment), or "MULTIPLE" (contradictory). This protocol achieved high agreement (Cohen’s $\kappa = 0.956$), allowing the authors to measure how often automated graders over-credit or under-credit models compared to a human reading of the same text.
Datasets
The audit covered a wide range of popular MLLM benchmarks:
- General/Reasoning: MMMU, ScienceQA, MathVista, MathVision.
- Industrial/Medical: MMAD (Industrial Anomaly Detection), OmniMedVQA, PMC-VQA.
- Hallucination: POPE.
The authors specifically used MMAD and PMC-VQA for their human-anchored adjudication because their formats (binary and 4-option) make human agreement highly reliable.
Key Insights and Ideas
- The "Snap" Artifact: Many benchmarks use a "no-threshold snap" (e.g.,
get_close_matches) that forces any string to the nearest option. On binary tasks, this heavily inflates scores by rewarding models that ramble without ever committing to an answer. - Bidirectional Errors: The same rule can help one model while hurting another. In Figure 3, the authors show that a snap rule adds 22 points to one model on MMAD but removes 19 points from another on PMC-VQA by overriding correct wordy answers with incorrect "nearest" letters.
- Likelihood is not a Silver Bullet: Scoring based on the model's logits (likelihood) is often seen as more "principled," but the authors found it is frequently biased by token-priors (e.g., a model preferring the token "A" regardless of the content).
- Reasoning Models aren't Immune: Even for models with 0% non-compliance (they always output a letter), the score can swing by 9.5 points based on whether the parser keeps the first or last regex match in a long chain-of-thought trace.
- The "Invisible" Problem: On leaderboards dominated by format-compliant frontier models, these grading artifacts are hidden. They only emerge when evaluating the "messier" open-source models used in specialized, air-gapped, or privacy-sensitive applications.
Key Results
The most striking result is the magnitude of the swing: swapping the parse-failure rule alone moves the accuracy score by up to 22 percentage points on identical model outputs (Figure 2).
On the MMAD benchmark, the "no-threshold snap" (the official rule) scored Llama-3.2-11B at 0.741, while an "abstention" rule (marking non-compliant answers wrong) scored the exact same outputs at 0.522.
Furthermore, the authors found that a single character of unreported regex in the ScienceQA benchmark was worth 29.5 points for one model, illustrating how fragile these metrics are to implementation details.
Limitations and Future Work
The authors acknowledge several boundaries:
- Model Size: The study focused on 7B–12B models; while the artifact is smaller on frontier models, it doesn't disappear, it just "migrates" from the fallback logic to the parser logic.
- Task Complexity: The human-anchored ground truth is strongest on binary and 4-option tasks. Scaling this to more complex open-ended tasks remains a challenge.
- Ranking: While the rules change absolute scores significantly, they did not find a "robust leaderboard reordering" for the very top-ranked models, though the gaps between mid-tier models were often entirely erased or reversed.
The paper concludes with a call for better reporting: benchmarks should report a non-compliance rate and provide accuracy scores under at least two different scoring rules to show the sensitivity of the results.
Summary updated