RAMP: Rethinking Multimodal Evaluation through Answer Readout Protocols

Jianye Zeng, Qiao Yikang, Duan Huang, Ziying SongarXiv 2026 · arXiv

TL;DR

The way we extract answers from Multimodal Large Language Models (MLLMs)—the "readout protocol"—can swing reported accuracy by over 20 percentage points on the same model and data. This paper introduces RAMP, a framework that proves evaluation scores are often as much a product of the interface (e.g., "yes/no" vs. "A/B") as they are of the model’s actual visual understanding.

What Problem Does This Paper Address?

When we evaluate an MLLM on a task like "Is there a dog in this image?", we don't just look at what the model "thinks." We have to map its internal probability scores (logits) to a discrete "Yes" or "No."

The problem is that there is no industry standard for this mapping. One researcher might look at the probability of the token "yes" vs. "no," while another might ask the model to choose between "A. yes" and "B. no." This paper addresses the readout sensitivity gap: the fact that a model's performance can appear to fluctuate wildly depending on these arbitrary choices, making fair model comparisons nearly impossible and hiding the model's true reasoning behavior.

Methods

The authors propose RAMP (Readout Aware Multimodal Protocol), a systematic framework to compare four primary "readout families":

  1. Bare-token: Comparing the raw logits of the lowercase tokens "yes" and "no."
  2. Surface-form: Expanding the candidate list to include various capitalizations (e.g., "Yes", "YES", "yes") to see if tokenization artifacts bias the score.
  3. Typed A/B: Forcing the model to choose between letter options (A vs. B) presented in the prompt.
  4. Typed-reversed: Swapping the labels (A: no, B: yes) to detect position bias.

Beyond aggregate accuracy, RAMP focuses on paired transitions—tracking exactly which questions flip from correct to incorrect when the interface changes. They also evaluate calibration (whether the model's confidence matches its accuracy) using Expected Calibration Error (ECE).

Datasets

The study evaluates three major MLLM families (SmolVLM, TinyLLaVA, and Qwen2.5-VL) across two key benchmarks:

  • POPE (Poll-Object-Probe-Evaluation): Specifically the adversarial subset (500 questions) designed to trigger object hallucinations.
  • ChartQA: A subset of 104 native binary (yes/no) questions focused on visual reasoning and data interpretation.

Key Insights and Ideas

  • The "Interface Tax": For smaller models, moving from a simple "yes/no" readout to a multiple-choice "A/B" interface acts like a massive performance penalty. SmolVLM-256M saw a staggering 20.2 percentage point drop in accuracy just by changing the readout.
  • Hidden Volatility: Net accuracy is a deceptive metric. In one experiment on ChartQA, a model showed a tiny +1.0 point net gain, but a "paired transition" analysis revealed that 76% of the individual answers actually changed. The model was essentially guessing differently, but the errors happened to cancel out in the final score.
  • Calibration Chaos: Readout protocols don't just change the answer; they change the model's "certainty." Typed A/B interfaces often significantly degrade a model's calibration, making its confidence scores less reliable for downstream applications.
  • Auditability is Key: The authors demonstrate that by saving raw first-token logits, they can "replay" evaluations offline with 100% agreement, suggesting that benchmarks should move toward storing raw scores rather than just final labels.

Key Results

  • Accuracy Swings: On POPE, SmolVLM-256M dropped from 71.8% (bare) to 51.6% (typed A/B). TinyLLaVA-1.1B dropped from 82.4% to 66.0%.
  • Model Robustness: Interestingly, the larger Qwen2.5-VL-3B was much more stable, actually gaining +1.8 points on the typed interface, suggesting that larger models may be better at handling the linguistic overhead of complex prompts.
  • Position Bias: Reversing A/B options exposed significant sensitivity; in one SmolVLM run, 401 out of 500 predictions changed when the order of "yes" and "no" was swapped.

Limitations and Future Work

  • Binary Focus: The current RAMP framework is optimized for binary (yes/no) questions. The authors acknowledge that extending this to open-ended generation or larger multiple-choice sets (e.g., A-E) is a necessary next step.
  • Prompt Engineering: While they tested A/B formatting, they didn't exhaustively test every possible prompt variation (e.g., "Think step by step").
  • Future Directions: The authors suggest that future MLLM benchmarks should adopt "readout-aware" reporting to ensure that a model's high score isn't just a lucky byproduct of a specific evaluation script.

Summary updated

Abstract

Multimodal large language models (MLLMs) are often evaluated by converting model outputs into discrete answers. The conversion protocol determines which scores become predictions and can strongly influence the reported results. We introduce RAMP (Readout Aware Multimodal Protocol), a systematic framework that compares direct yes/no token logits, surface-form scores, and typed or reversed A/B interfaces under fixed model checkpoints and paired visual questions. On 500 POPE adversarial questions, typed versus bare readout changes accuracy by $-20.2$ percentage points for SmolVLM-256M, $-16.4$ for TinyLLaVA-1.1B, and $+1.8$ for Qwen2.5-VL-3B. Paired transitions reveal extensive prediction changes even when the net accuracy difference is small: 79 of 104 binary ChartQA items change correctness for SmolVLM-500M, despite a $+1.0$-point net difference. We further analyze prompt wording, candidate forms, option position, cross-task effects, and calibration. An independent offline replay reproduces 16,140 first-token decisions from saved raw scores with exact agreement. Our findings establish answer readout as a measurable component of multimodal evaluation and provide a reproducible protocol for analyzing model comparisons.
BibTeX
@misc{zeng2026ramp,
  author = {Jianye Zeng and Qiao Yikang and Duan Huang and Ziying Song},
  title = {{RAMP: Rethinking Multimodal Evaluation through Answer Readout Protocols}},
  year = {2026},
  eprint = {2611.20653},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.20653},
}
Figure