TL;DR
Researchers have developed Belief-Grouped Policy Optimization (BGPO), a post-training method that decomposes aggregate probabilities into latent "belief groups." By distinguishing whether a 50/50 prediction stems from a split population or individual uncertainty, BGPO significantly improves the calibration and interpretability of decision models across text and image tasks.
What Problem Does This Paper Address?
When a decision model predicts a 50% probability for "Option A" and 50% for "Option B," it is technically "calibrated" if those options occur equally often in the real world. However, this single number hides a critical distinction:
- Disagreement: Half the population is 100% sure it’s A, and the other half is 100% sure it’s B.
- Uncertainty: Every single person is torn and thinks it’s a coin flip between A and B.
Standard training methods (like SFT or RLHF) pool these two scenarios into a single distribution, treating disagreement as noise. This is problematic for "pluralistic alignment"—the goal of making AI reflect the diverse perspectives of a human population. If we don't know why people disagree, we can't identify which perspective applies to a specific user or determine if we simply need more evidence to resolve the ambiguity.
Methods
The authors introduce Belief-Grouped Policy Optimization (BGPO). Instead of outputting one distribution per question, the model is trained to act as a population simulator in two stages:
- Belief List Generation: The model generates a list of latent "beliefs" (e.g., "a viewer reminded of home"), estimates their prevalence (shares), and predicts which demographic labels (like "Chinese-language speaker") are likely to hold them.
- Belief-Specific Answering: For each belief in the list, the model generates a specific choice distribution.
The Secret Sauce: Bayesian Credit Assignment During training, BGPO uses Bayes’ rule to calculate "responsibility credit." If an annotator chose "Sadness," the training signal is weighted toward the belief groups that were most likely to produce that choice. This allows the model to learn distinct perspectives even when they are mixed within the same demographic label.
To ensure the model doesn't miss rare perspectives, the authors use Posterior Lists. The model is prompted to propose beliefs after seeing a few real-world choices, helping it "discover" minority views that it might otherwise ignore.
Datasets
The model was evaluated on five diverse benchmarks:
- Text: OpinionQA (Pew surveys), GlobalOpinionQA (international surveys), and DICES (safety ratings).
- Image: ArtELingo (emotions elicited by paintings) and PARA (photo aesthetics).
- CLEVR-Belief (New): A synthetic image benchmark introduced in this paper where "annotators" follow known, hidden rules (e.g., "always pick the leftmost object"). This allows the researchers to measure exactly how well the model recovers true latent beliefs.
Key Insights and Ideas
- Labels are Evidence, Not Ground Truth: Traditional methods often condition directly on demographic labels (e.g., "As a Republican..."). BGPO treats these labels as noisy evidence of a deeper latent belief, acknowledging that people within the same group can interpret the same question differently.
- The Cost of Pooling: The paper mathematically demonstrates that pooling different beliefs into one distribution costs exactly the "Mutual Information" between the choice and the belief. BGPO recovers this lost information.
- Grouping Prevents Erasure: By grouping advantages by belief during RL, the model prevents common beliefs from "washing out" rare ones. This ensures that minority perspectives receive a strong enough gradient to be learned.
Key Results
BGPO consistently outperformed baselines (including SFT, PRDO, and GRPO) across all metrics:
- Lower Divergence: It achieved the lowest Jensen–Shannon Divergence (JSD) at the population, group, and worst-case levels. On text tasks, it reduced conditional divergence by up to 44% compared to the strongest baseline.
- Belief Recovery: In the synthetic CLEVR-Belief task, BGPO achieved near-perfect agreement with true underlying rules (Adjusted Rand Index of ~0.99), whereas standard label-conditioned models struggled (~0.37).
- Zero-Shot Performance: Even when initialized from smaller models (Qwen3.5-2B), BGPO often outperformed larger models trained with standard objectives.
Limitations and Future Work
- Closed-Ended Options: Currently, BGPO is designed for tasks with a fixed set of options (multiple choice). Scoring "beliefs" against open-ended, free-text responses remains a challenge.
- Computational Overhead: Generating multiple belief lists and sampling answers for each belief increases the token budget during training compared to standard single-prediction models.
- Inference Simplicity: While the training is complex, at inference, the model only receives the question. Future work could explore using the recovered belief structure to allow users to "nudge" the model toward specific perspectives.
Summary updated