ConfProBench: A Confidence Evaluation Benchmark for MLLM-Based Process Judges

Yue Zhou, Yi Chang, Yuan Wu

Abstract

Multimodal Large Language Models (MLLMs) are increasingly deployed as "Process Judges" (MPJs) to evaluate individual steps in complex reasoning tasks. However, the reliability of the confidence scores these judges provide remains largely unexamined. We introduce ConfProBench, a comprehensive benchmark designed to assess the robustness, sensitivity, and calibration of MPJ confidence. By testing 14 state-of-the-art models against adversarial perturbations, we find that even top-tier models exhibit significant fragility in their confidence estimations, particularly under syntactic transformations and when failing to detect reasoning errors.

Introduction

As MLLMs tackle multi-step reasoning in domains like calculus or organic chemistry, the need for reliable automated grading at the step level has grown. Process Judges (MPJs) fulfill this role by providing both a correctness judgment and a confidence score for each reasoning step. Existing benchmarks primarily measure classification accuracy, neglecting whether the judge's "certainty" is well-founded. ConfProBench addresses this gap by evaluating if a judge's confidence is robust to irrelevant changes, sensitive to actual errors, and well-calibrated to its true accuracy.

Methods and Dataset Construction

ConfProBench evaluates MPJs using three core dimensions: Confidence Robustness Score (CRS), Confidence Sensitivity Score (CSS), and Confidence Calibration Score (CCS). To rigorously test these dimensions, the researchers developed a pipeline to generate adversarial perturbations that should not affect a reliable judge's confidence but often do.

The ConfProBench construction pipeline, illustrating the use of adversarial perturbations to evaluate judge reliability.

The benchmark consists of 1,200 scientific problems across Math, Physics, Chemistry, and Biology, spanning middle school to competition levels. The evaluation includes three types of perturbations:

  • Synonym Substitution: Replacing non-technical words with equivalents.
  • Syntactic Transformation: Changing sentence structure (e.g., active to passive voice).
  • Image Perturbation: Adding noise, rotating, or scaling visual inputs.

The Process Judging Task

In the context of ConfProBench, a "Process Judge" must analyze a multi-step reasoning chain and provide a confidence score for the validity of each step. A high-quality judge should maintain stable confidence despite stylistic changes and show a marked drop in confidence when encountering a reasoning flaw.

An example of an MLLM acting as a Process Judge, assigning confidence scores to individual reasoning steps.

Key Insights and Results

The study evaluated 14 models, including proprietary leaders like GPT-4o and Gemini, alongside open-source models like Qwen2.5-VL.

  • Syntactic Fragility: Models are surprisingly sensitive to sentence structure. Syntactic transformations caused the most significant drops in robustness, suggesting that MPJs are still overly influenced by the "style" of a reasoning step rather than its logical substance.
  • The "Thinking" Trade-off: Enabling "thinking" modes (e.g., Gemini 2.5 Flash) significantly improves a model's sensitivity to errors and its robustness. However, it can actually hurt calibration, making the model's confidence scores less representative of its actual probability of being correct.
  • Scale vs. Reliability: Larger models generally offer better calibration (GPT-4o was the "Calibration King"), but they are not necessarily more robust to adversarial attacks than their smaller counterparts.
  • Overconfidence in Errors: Across almost all models, calibration is much worse for incorrect steps. Models are generally "overconfident" when they fail to spot a reasoning error.

Conclusion and Future Work

ConfProBench reveals that current MLLM Process Judges struggle with reliable confidence estimation. While models like Gemini-2.5-flash and GPT-4o lead in specific metrics, there remains a significant gap between current performance and theoretical reliability. Future work should focus on aligning model certainty with human intuition and extending these evaluations to safety-critical domains like medicine and law, where misplaced confidence could have dire consequences.

Summary updated

Abstract

Reasoning is the critical capability of multimodal large language models (MLLMs) to solve complex multimodal tasks, and judging the correctness of reasoning steps is crucial to improving this capability. Recently, MLLM-based process judges (MPJs) have been widely used to judge the correctness of reasoning steps in multimodal reasoning tasks. Therefore, evaluating the capability of MPJs is crucial for identifying their limitations and guiding future improvements. However, existing benchmarks for MPJs primarily focus on evaluating capabilities such as step correctness classification and reasoning process search, while overlooking a critical dimension: whether the confidence scores produced by MPJs at the step level are reliable. To fill this gap, we propose ConfProBench, the first comprehensive benchmark designed to systematically evaluate the reliability of step-level confidence scores generated by MPJs. This benchmark constructs three types of adversarially perturbed reasoning steps: Synonym Substitution, Syntactic Transformation, and Image Perturbation, to evaluate the robustness of MPJs' confidence under perturbations. Furthermore, we propose three novel evaluation metrics: Confidence Robustness Score (CRS), Confidence Sensitivity Score (CSS), and Confidence Calibration Score (CCS), which are designed to capture three complementary aspects of MPJs' confidence—robustness, sensitivity, and calibration. We evaluate 14 state-of-the-art MLLMs, including both proprietary and open-source models. Through extensive experiments, we reveal limitations in existing MPJs’ confidence performance and provide competitive baselines, thereby paving the way for future research in this field. Our dataset is provided in the supplementary materials.
BibTeX
@misc{zhou2025confprobench,
  author = {Yue Zhou and Yi Chang and Yuan Wu},
  title = {{ConfProBench: A Confidence Evaluation Benchmark for MLLM-Based Process Judges}},
  year = {2025},
  eprint = {2508.04576},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2508.04576},
}
Figure