Abstract
Multimodal Large Language Models (MLLMs) are increasingly deployed as "Process Judges" (MPJs) to evaluate individual steps in complex reasoning tasks. However, the reliability of the confidence scores these judges provide remains largely unexamined. We introduce ConfProBench, a comprehensive benchmark designed to assess the robustness, sensitivity, and calibration of MPJ confidence. By testing 14 state-of-the-art models against adversarial perturbations, we find that even top-tier models exhibit significant fragility in their confidence estimations, particularly under syntactic transformations and when failing to detect reasoning errors.
Introduction
As MLLMs tackle multi-step reasoning in domains like calculus or organic chemistry, the need for reliable automated grading at the step level has grown. Process Judges (MPJs) fulfill this role by providing both a correctness judgment and a confidence score for each reasoning step. Existing benchmarks primarily measure classification accuracy, neglecting whether the judge's "certainty" is well-founded. ConfProBench addresses this gap by evaluating if a judge's confidence is robust to irrelevant changes, sensitive to actual errors, and well-calibrated to its true accuracy.
Methods and Dataset Construction
ConfProBench evaluates MPJs using three core dimensions: Confidence Robustness Score (CRS), Confidence Sensitivity Score (CSS), and Confidence Calibration Score (CCS). To rigorously test these dimensions, the researchers developed a pipeline to generate adversarial perturbations that should not affect a reliable judge's confidence but often do.
The benchmark consists of 1,200 scientific problems across Math, Physics, Chemistry, and Biology, spanning middle school to competition levels. The evaluation includes three types of perturbations:
- Synonym Substitution: Replacing non-technical words with equivalents.
- Syntactic Transformation: Changing sentence structure (e.g., active to passive voice).
- Image Perturbation: Adding noise, rotating, or scaling visual inputs.
The Process Judging Task
In the context of ConfProBench, a "Process Judge" must analyze a multi-step reasoning chain and provide a confidence score for the validity of each step. A high-quality judge should maintain stable confidence despite stylistic changes and show a marked drop in confidence when encountering a reasoning flaw.
Key Insights and Results
The study evaluated 14 models, including proprietary leaders like GPT-4o and Gemini, alongside open-source models like Qwen2.5-VL.
- Syntactic Fragility: Models are surprisingly sensitive to sentence structure. Syntactic transformations caused the most significant drops in robustness, suggesting that MPJs are still overly influenced by the "style" of a reasoning step rather than its logical substance.
- The "Thinking" Trade-off: Enabling "thinking" modes (e.g., Gemini 2.5 Flash) significantly improves a model's sensitivity to errors and its robustness. However, it can actually hurt calibration, making the model's confidence scores less representative of its actual probability of being correct.
- Scale vs. Reliability: Larger models generally offer better calibration (GPT-4o was the "Calibration King"), but they are not necessarily more robust to adversarial attacks than their smaller counterparts.
- Overconfidence in Errors: Across almost all models, calibration is much worse for incorrect steps. Models are generally "overconfident" when they fail to spot a reasoning error.
Conclusion and Future Work
ConfProBench reveals that current MLLM Process Judges struggle with reliable confidence estimation. While models like Gemini-2.5-flash and GPT-4o lead in specific metrics, there remains a significant gap between current performance and theoretical reliability. Future work should focus on aligning model certainty with human intuition and extending these evaluations to safety-critical domains like medicine and law, where misplaced confidence could have dire consequences.
Summary updated