What a Calibrated Probability Hides: Belief-Level Post-Training for Decision Models

Xinye Li, Zirui Song, Huaxing Liu, Xiang Wang, Shuai Li, Zixiang Xu, Yinjun Liu, Zhewen Zhang, Xiaojun Chang, Xiuying ChenarXiv 2026 · arXiv

TL;DR

Researchers have developed Belief-Grouped Policy Optimization (BGPO), a post-training method that decomposes aggregate probabilities into latent "belief groups." By distinguishing whether a 50/50 prediction stems from a split population or individual uncertainty, BGPO significantly improves the calibration and interpretability of decision models across text and image tasks.

What Problem Does This Paper Address?

When a decision model predicts a 50% probability for "Option A" and 50% for "Option B," it is technically "calibrated" if those options occur equally often in the real world. However, this single number hides a critical distinction:

  1. Disagreement: Half the population is 100% sure it’s A, and the other half is 100% sure it’s B.
  2. Uncertainty: Every single person is torn and thinks it’s a coin flip between A and B.

Standard training methods (like SFT or RLHF) pool these two scenarios into a single distribution, treating disagreement as noise. This is problematic for "pluralistic alignment"—the goal of making AI reflect the diverse perspectives of a human population. If we don't know why people disagree, we can't identify which perspective applies to a specific user or determine if we simply need more evidence to resolve the ambiguity.

Figure 1: Two sources of uncertainty behind the same 50/50 prediction. Top: Two equally sized groups favor different emotions, so identifying which interpretation fits a viewer helps predict their response. Bottom: Each viewer is equally torn between contentment and sadness, so gathering more evidence may help resolve their uncertainty.

Methods

The authors introduce Belief-Grouped Policy Optimization (BGPO). Instead of outputting one distribution per question, the model is trained to act as a population simulator in two stages:

  1. Belief List Generation: The model generates a list of latent "beliefs" (e.g., "a viewer reminded of home"), estimates their prevalence (shares), and predicts which demographic labels (like "Chinese-language speaker") are likely to hold them.
  2. Belief-Specific Answering: For each belief in the list, the model generates a specific choice distribution.

The Secret Sauce: Bayesian Credit Assignment During training, BGPO uses Bayes’ rule to calculate "responsibility credit." If an annotator chose "Sadness," the training signal is weighted toward the belief groups that were most likely to produce that choice. This allows the model to learn distinct perspectives even when they are mixed within the same demographic label.

To ensure the model doesn't miss rare perspectives, the authors use Posterior Lists. The model is prompted to propose beliefs after seeing a few real-world choices, helping it "discover" minority views that it might otherwise ignore.

Figure 2: BGPO training on a single question. (1) A shared policy generates 𝐿belief lists from the question and 𝐻posterior lists conditioned on observed annotations. (2) It samples 𝑛answers per belief. (3) Bayes’ rule assigns soft credit for held-out annotations to beliefs, yielding belief-specific answer scores. List scores use all held-out annotations, and posterior scores use observed annotations. (4) Within-group advantages update the shared policy.

Datasets

The model was evaluated on five diverse benchmarks:

  • Text: OpinionQA (Pew surveys), GlobalOpinionQA (international surveys), and DICES (safety ratings).
  • Image: ArtELingo (emotions elicited by paintings) and PARA (photo aesthetics).
  • CLEVR-Belief (New): A synthetic image benchmark introduced in this paper where "annotators" follow known, hidden rules (e.g., "always pick the leftmost object"). This allows the researchers to measure exactly how well the model recovers true latent beliefs.

Key Insights and Ideas

  • Labels are Evidence, Not Ground Truth: Traditional methods often condition directly on demographic labels (e.g., "As a Republican..."). BGPO treats these labels as noisy evidence of a deeper latent belief, acknowledging that people within the same group can interpret the same question differently.
  • The Cost of Pooling: The paper mathematically demonstrates that pooling different beliefs into one distribution costs exactly the "Mutual Information" between the choice and the belief. BGPO recovers this lost information.
  • Grouping Prevents Erasure: By grouping advantages by belief during RL, the model prevents common beliefs from "washing out" rare ones. This ensures that minority perspectives receive a strong enough gradient to be learned.

Key Results

BGPO consistently outperformed baselines (including SFT, PRDO, and GRPO) across all metrics:

  • Lower Divergence: It achieved the lowest Jensen–Shannon Divergence (JSD) at the population, group, and worst-case levels. On text tasks, it reduced conditional divergence by up to 44% compared to the strongest baseline.
  • Belief Recovery: In the synthetic CLEVR-Belief task, BGPO achieved near-perfect agreement with true underlying rules (Adjusted Rand Index of ~0.99), whereas standard label-conditioned models struggled (~0.37).
  • Zero-Shot Performance: Even when initialized from smaller models (Qwen3.5-2B), BGPO often outperformed larger models trained with standard objectives.
Quantitative results on text benchmarks.

Limitations and Future Work

  • Closed-Ended Options: Currently, BGPO is designed for tasks with a fixed set of options (multiple choice). Scoring "beliefs" against open-ended, free-text responses remains a challenge.
  • Computational Overhead: Generating multiple belief lists and sampling answers for each belief increases the token budget during training compared to standard single-prediction models.
  • Inference Simplicity: While the training is complex, at inference, the model only receives the question. Future work could explore using the recovered belief structure to allow users to "nudge" the model toward specific perspectives.

Summary updated

Abstract

A calibrated decision model can predict how often each answer is chosen. The same prediction can reflect disagreement among people who are confident in their choices or uncertainty among people who share a belief. Disagreement calls for identifying whose perspective applies, while uncertainty calls for gathering more evidence. Standard post-training scores a single prediction against one observed choice at a time, learning the pooled distribution without distinguishing these two sources of uncertainty. We introduce Belief-Grouped Policy Optimization (BGPO). The policy proposes a list of beliefs, estimates their prevalence, and predicts a choice distribution for each. During training, BGPO uses Bayes' rule to estimate how likely each belief was to generate an observed choice. These estimates weight the choice's contribution to each belief's log-score reward. To discover rare beliefs, the policy also proposes lists after seeing a few observed choices and retains them only if they predict held-out choices well. At inference, the policy uses only the question and combines the belief-specific distributions by prevalence to produce a calibrated aggregate prediction. On benchmarks covering opinions, conversation safety, emotions elicited by artwork, and photo aesthetics, BGPO achieves the lowest conditional, worst-group, and overall Jensen-Shannon divergence across model initializations. It improves conditional divergence on both text and image tasks. On a synthetic image task with known beliefs, BGPO recovers the underlying groups more accurately than baselines trained with belief labels.
BibTeX
@misc{li2026what,
  author = {Xinye Li and Zirui Song and Huaxing Liu and Xiang Wang and Shuai Li and Zixiang Xu and Yinjun Liu and Zhewen Zhang and Xiaojun Chang and Xiuying Chen},
  title = {{What a Calibrated Probability Hides: Belief-Level Post-Training for Decision Models}},
  year = {2026},
  eprint = {2611.25168},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.25168},
}
Figure