TL;DR
The way we extract answers from Multimodal Large Language Models (MLLMs)—the "readout protocol"—can swing reported accuracy by over 20 percentage points on the same model and data. This paper introduces RAMP, a framework that proves evaluation scores are often as much a product of the interface (e.g., "yes/no" vs. "A/B") as they are of the model’s actual visual understanding.
What Problem Does This Paper Address?
When we evaluate an MLLM on a task like "Is there a dog in this image?", we don't just look at what the model "thinks." We have to map its internal probability scores (logits) to a discrete "Yes" or "No."
The problem is that there is no industry standard for this mapping. One researcher might look at the probability of the token "yes" vs. "no," while another might ask the model to choose between "A. yes" and "B. no." This paper addresses the readout sensitivity gap: the fact that a model's performance can appear to fluctuate wildly depending on these arbitrary choices, making fair model comparisons nearly impossible and hiding the model's true reasoning behavior.
Methods
The authors propose RAMP (Readout Aware Multimodal Protocol), a systematic framework to compare four primary "readout families":
- Bare-token: Comparing the raw logits of the lowercase tokens "yes" and "no."
- Surface-form: Expanding the candidate list to include various capitalizations (e.g., "Yes", "YES", "yes") to see if tokenization artifacts bias the score.
- Typed A/B: Forcing the model to choose between letter options (A vs. B) presented in the prompt.
- Typed-reversed: Swapping the labels (A: no, B: yes) to detect position bias.
Beyond aggregate accuracy, RAMP focuses on paired transitions—tracking exactly which questions flip from correct to incorrect when the interface changes. They also evaluate calibration (whether the model's confidence matches its accuracy) using Expected Calibration Error (ECE).
Datasets
The study evaluates three major MLLM families (SmolVLM, TinyLLaVA, and Qwen2.5-VL) across two key benchmarks:
- POPE (Poll-Object-Probe-Evaluation): Specifically the adversarial subset (500 questions) designed to trigger object hallucinations.
- ChartQA: A subset of 104 native binary (yes/no) questions focused on visual reasoning and data interpretation.
Key Insights and Ideas
- The "Interface Tax": For smaller models, moving from a simple "yes/no" readout to a multiple-choice "A/B" interface acts like a massive performance penalty. SmolVLM-256M saw a staggering 20.2 percentage point drop in accuracy just by changing the readout.
- Hidden Volatility: Net accuracy is a deceptive metric. In one experiment on ChartQA, a model showed a tiny +1.0 point net gain, but a "paired transition" analysis revealed that 76% of the individual answers actually changed. The model was essentially guessing differently, but the errors happened to cancel out in the final score.
- Calibration Chaos: Readout protocols don't just change the answer; they change the model's "certainty." Typed A/B interfaces often significantly degrade a model's calibration, making its confidence scores less reliable for downstream applications.
- Auditability is Key: The authors demonstrate that by saving raw first-token logits, they can "replay" evaluations offline with 100% agreement, suggesting that benchmarks should move toward storing raw scores rather than just final labels.
Key Results
- Accuracy Swings: On POPE, SmolVLM-256M dropped from 71.8% (bare) to 51.6% (typed A/B). TinyLLaVA-1.1B dropped from 82.4% to 66.0%.
- Model Robustness: Interestingly, the larger Qwen2.5-VL-3B was much more stable, actually gaining +1.8 points on the typed interface, suggesting that larger models may be better at handling the linguistic overhead of complex prompts.
- Position Bias: Reversing A/B options exposed significant sensitivity; in one SmolVLM run, 401 out of 500 predictions changed when the order of "yes" and "no" was swapped.
Limitations and Future Work
- Binary Focus: The current RAMP framework is optimized for binary (yes/no) questions. The authors acknowledge that extending this to open-ended generation or larger multiple-choice sets (e.g., A-E) is a necessary next step.
- Prompt Engineering: While they tested A/B formatting, they didn't exhaustively test every possible prompt variation (e.g., "Think step by step").
- Future Directions: The authors suggest that future MLLM benchmarks should adopt "readout-aware" reporting to ensure that a model's high score isn't just a lucky byproduct of a specific evaluation script.
Summary updated