Language-model agents increasingly work side by side on research and search tasks. They often share a scratchpad, where one agent's verified fact saves another agent a search and one agent's careless claim sends another down a wrong path. The easy ways to cut costs are to skip reading the source, to post a snippet as if it were verified, and to take others' posts without contributing any. These are hidden-effort problems of the kind economists call moral hazard. Mechanism Design Is Not Enough shows two things relevant here. When effort cannot be observed, contracts cannot close the resulting welfare gap. Frontier models also behave like purely self-interested agents by default: their fitted prosociality weight $\lambda$, the weight they place on others' payoff, is 0. The fix that paper points to is an intrinsic prosocial weight. So far that weight has only been studied in matrix games and fishery simulations, where the reward is scripted and cannot be checked against ground truth.
This proposal builds a mixed-motive game on top of multi-hop question answering, where a question can only be answered by chaining facts across documents. The game has two properties the social-dilemma benchmarks lack. First, every contribution to the shared scratchpad can be checked against gold intermediate answers. Second, cooperation pays off in exactly the place single-agent retrieval fails: the "bridge" fact that one question needs and another question already found. We train agents with reinforcement learning on a utility that weights their own reward against their peers'. We then ask whether the learned disposition carries over, both to unseen retrieval domains and to classic social dilemmas the agents never trained on. The central bet comes from the reasoning-transfer literature: skills learned under verifiable rewards transfer where skills learned under soft rewards do not. We test whether that holds for social dispositions as well as for problem-solving skills.
Core Claim
Prosociality learned with reinforcement learning in a verifiable shared-evidence game should do three things. It should reduce free-riding and unread "snippet posting" in multi-agent retrieval. It should resist exploitation by shirking peers. It should also raise the prosociality weight that can be estimated from the agent's play in held-out social games, beyond what persona prompting or training on scripted game rewards achieves. The specific new ingredient is not "add a social weight to a retrieval team." It is training against a mixed population of peers with different social weights, in an environment where effort is partly hidden but outcomes are checkable. We expect that training to teach conditional cooperation rather than blanket generosity.
Why This Intersection, and What Already Exists
The multi-hop reasoning gap motivates the environment. Iterative methods such as IRCoT interleave retrieval with reasoning because the second-hop query cannot be formed until the bridge fact is known. MuSiQue builds questions by composing single-hop questions so that shortcuts do not work. As a result, many questions in that dataset share single-hop sub-questions, which is exactly the structure a shared scratchpad can use.
Two papers constrain the design. Before Reasoning Can Fail finds that agents frequently answer from search snippets without ever calling a read tool, and that adding hidden thinking tokens makes this worse. FrugalRAG uses reinforcement learning with length penalties to cut searches from 5.99 to 2.05 on HotpotQA. Put together, these suggest that a private cost on reading pushes agents toward the discipline failure. In a shared setting that failure is also a harm to others. A prosocial weight is the natural counterweight, because the cost is paid privately while the damage from a bad post spreads to peers.
Multi-agent retrieval systems supply the architecture and set the contrast. PRISMA splits multi-hop QA across Planner, Retriever, Inspector, Solver and Memoizer agents trained on a shared task reward. Because all its agents optimize one objective, there are no conflicting incentives and nothing for a social weight to do. Our agents are peers with their own questions and their own costs, so the incentive conflict is real rather than imposed.
Several papers supply the social-utility machinery, and each leaves a gap. Mechanism Design Is Not Enough provides the $\lambda$ formulation and the quantal-response-equilibrium procedure for estimating $\lambda$ from observed play, a model of choice in which agents pick better options more often but not always. We reuse that estimation procedure as a transfer probe. Learning Roles with Emergent Social Value Orientations learns reward-sharing roles in grid worlds, with no language and no transfer tests. AgentElect assigns social personas through prompts and does not train them.
On the transfer side, HILA trains a policy over solve, endorse and defer decisions with GRPO, a reinforcement learning method that scores each sample relative to a group of samples for the same prompt, and shows the policy transfers across reasoning benchmarks. It treats collaboration as cost minimization, however, not as a weighting of other agents' payoffs. Reasoning Vectors isolates what RL training added by subtracting weights, $\theta_{\text{GRPO}} - \theta_{\text{SFT}}$, and transfers it into other models. We borrow that subtraction as a secondary analysis tool.
To our reading of the file, no work connects social value orientation to multi-hop retrieval at all.
Research Program
The game. We build the game, which we call Shared Bridge, from MuSiQue. We group questions into clusters of $N=4$ that share at least one single-hop sub-question, using MuSiQue's released decompositions. Unanswerable contrast items from MuSiQue-Full go into the clusters so that posting a wrong "bridge" fact has a cost. Each agent receives one question and access to five tools: search (returns snippets), read (returns a full paragraph and costs $c_r$), post (writes a fact plus a cited paragraph ID to a shared board and costs $c_p$), view-board, and answer. Agent $i$'s own reward is
$$r_i = \mathrm{F1}(\hat a_i, a_i) - c_r\, n^{\text{read}}_i - c_p\, n^{\text{post}}_i,$$
where $\hat a_i$ is its answer, $a_i$ the gold answer, and $n^{\text{read}}_i, n^{\text{post}}_i$ its tool counts. Its training utility is
$$U_i = r_i + \lambda_i \frac{1}{N-1}\sum_{j\neq i} r_j,$$
where $\lambda_i$ is agent $i$'s social weight. $\lambda_i = 0$ is the self-interested case and $\lambda_i = 1$ weights the group equally with oneself.
Effort is partly hidden in a deliberate way. Peers see a post's text and citation, but not whether the poster actually read the cited paragraph or checked that it supports the claim. The trainer, however, can check posts against the gold sub-answers. That check determines outcomes only indirectly, through the peers' final correctness. It never enters the reward as a direct bonus, so we are not simply paying agents for good posts.
Training. The backbone is Qwen-2.5-7B, which HILA also uses. Each agent is conditioned on its own $\lambda_i$, written into the prompt as a number, and trained by GRPO. Following FrugalRAG, a short supervised warm-up precedes RL. The key design choice is the peer population. Within a cluster, peers are drawn from a mix of self-interested ($\lambda = 0$) agents, prosocial agents, and scripted shirkers that post snippet-derived claims without reading. The aim is to learn cooperation that responds to the evidence behind each post: trusting posts whose citations check out when read, and falling back to one's own reading when they do not. We expect this to matter because prosocial agents are known to be exploited by self-interested ones (Mechanism Design Is Not Enough).
Analysis. We fit per-agent $\lambda$ using the paper's quantal-response-equilibrium procedure on TableGames and GovSimContract, both before and after training. We also measure four behaviors in the retrieval game. The first is the rate of posts made without a prior read of the cited paragraph, our measure of "snippet posting." The second is board reuse, meaning how often a posted fact is actually used by the agent whose question needs it. The third is total reads per correct answer across the cluster. The fourth is the payoff gap between prosocial agents and shirkers, a measure of exploitation. As a secondary analysis, we compute a social weight direction, the weights of the $\lambda$-trained model minus those of the $\lambda=0$-trained model, in the style of Reasoning Vectors. We then test whether adding this direction to the base model moves its fitted $\lambda$ in games. That would tell us whether the disposition sits in a separable part of the weights or is entangled with retrieval skill.
Baselines
The naive reference point is the untrained Qwen-2.5-7B in the same Shared Bridge harness with no social instruction. Given the finding that frontier models default to $\lambda = 0$, we expect it to free-ride and snippet-post. It anchors every metric and sets the floor for fitted $\lambda$ in the games.
The first strong competitor is the same model given an explicit prosocial persona prompt, in the style of AgentElect's social personas (Evaluating Cooperation in LLM Social Groups). It is the cheapest way to get cooperation. It runs on identical clusters with the same tool costs, and we tune the persona wording over the same number of validation trials as our RL hyperparameters. Beating it means, at minimum, fewer unread posts, higher accuracy across the cluster at equal or lower reads, and a higher fitted $\lambda$ on TableGames.
The strongest retrieval competitor is a FrugalRAG-style single agent (FrugalRAG), trained with the same GRPO budget and the same number of training questions on the self-only reward. It is deployed either with no board or with the board available. FrugalRAG reports 68.47% model-based evaluation accuracy on HotpotQA with 2.05 searches, against 64.20% for ReAct. We reproduce that setting as a sanity check on our training pipeline. In Shared Bridge, beating this baseline means higher accuracy across the cluster per unit of retrieval cost. On single-agent MuSiQue and HotpotQA, "not losing" to it is the requirement.
The strongest structured multi-agent competitor is PRISMA-style shared team reward (PRISMA), in which every agent optimizes the cluster's total reward. In our notation this is homogeneous self-play at $\lambda = 1$. PRISMA reports 23.6% exact match on 3-hop MuSiQue against 8.9% for a prior baseline. That is a reference for a different architecture, not for our game, and no reported result exists on Lacuna for any shared-board, mixed-motive multi-hop setting.
The ablation that isolates the key ingredient keeps everything identical except the peer population: homogeneous prosocial self-play versus the heterogeneous population with shirkers. The proposal's claim rests on the heterogeneous version matching homogeneous self-play when all peers cooperate, and clearly beating it on exploitation and on transfer to games. A second ablation adds a direct reward for posts that match gold sub-answers, to check that we are not just rediscovering reward shaping.
For the game-transfer tests there are two reference points. One is the TableGames result in Mechanism Design Is Not Enough: a 60–68 percentage point gap in utilitarian efficiency that contracts, even code-enforced ones, leave open. The other is its GovSimContract result: +119.6 units in total gains when moving from selfish to prosocial agents. Our models are compared on those same environments, run the same way.
Why not just…?
Why not just prompt the agents to be prosocial? The evidence in the file suggests prompting does not change what models actually optimize. Frontier models fit $\lambda = 0$ even with natural-language contracts (Mechanism Design Is Not Enough), and in GovSim, prompted cooperative groups were still pulled toward over-extraction without governance structure (Evaluating Cooperation in LLM Social Groups). Prompts may still reduce free-riding when reading is cheap. So we sweep the read cost $c_r$ and report where, if anywhere, prompting matches training. If it matches everywhere, the claim fails.
Why not just give everyone a shared team reward, as PRISMA does? In a closed lab team this is a real threat. It may well match our method in-domain, and we expect it to. The difference should appear in two places. The first is exploitation: team-reward agents trained only with cooperative peers have never seen a shirker and should trust bad posts. The second is transfer: many deployments have no shared objective to impose, since the agents belong to different parties. The homogeneous-versus-heterogeneous ablation settles this directly. If homogeneous training transfers just as well and resists shirkers just as well, we will report that the population design adds nothing.
Why not just enforce a rule, like READ-GATE's "no answer without a read"? Before Reasoning Can Fail shows such a rule removes zero-read failures, and we include a rule requiring a read before any post as an extra condition. But the rule checks that a read happened, not that the cited paragraph supports the claim. That second kind of effort stays hidden, which is exactly the setting where the file's results say rules cannot close the gap. We measure how often gated agents post claims their cited paragraph does not support. If the rule alone drives that rate near zero, the social-weight contribution is limited to cases the rule cannot see, and we will say so.
Why not just train on GovSim or TableGames directly? If the goal were only prosociality in games, that would be the obvious route, and a GovSim-trained model is a required comparison. The question here is which training signal produces a disposition that works in both kinds of setting. We run the transfer in both directions: game-trained models tested in Shared Bridge, and Shared Bridge-trained models tested in the games. If game training wins in both directions, the verifiable-reward hypothesis is false for social dispositions. That is a useful negative result, and it would bear on the open problem of verifiable rewards in social settings raised in the reasoning-transfer literature.
Proposed Experiments and Decisive Checks
In-domain, we evaluate on held-out Shared Bridge clusters built from MuSiQue's dev split. We report accuracy across the cluster (exact match and F1), reads per correct answer, unread-post rate, board reuse, and the prosocial-versus-shirker payoff gap. Results are shown across $\lambda \in \{0, 0.5, 1\}$ and three read costs.
For out-of-domain retrieval, we build Shared Bridge clusters from 2WikiMultiHopQA, which has structured relations that make shared bridge entities easy to find, and from Bamboogle. Where they can be obtained, we also use PRISMA's five domain datasets (chemistry, food, games, geography, music). This tests whether the cooperation policy survives unfamiliar corpora.
For social transfer, we run TableGames, GovSimContract and AgentElect unchanged. We report fitted $\lambda$, utilitarian efficiency, total gains and resource survival time, all compared with the numbers reported in Mechanism Design Is Not Enough and Evaluating Cooperation in LLM Social Groups. Because prosocial governance in GovSim can raise inequality, we also report the Gini coefficient, so that a welfare gain bought at the cost of greater inequality is visible.
As a guard against general harm, we evaluate single-agent MuSiQue and HotpotQA with the board disabled, plus GSM8K and BigBenchHard. Reasoning Vectors shows RL-derived weight changes can shift these scores, so a loss here would be a real cost.
The decisive check is a single comparison: heterogeneous-population training versus homogeneous team-reward training, with equal compute, evaluated on two outcomes. One is the payoff an agent loses to scripted shirkers in held-out Shared Bridge clusters. The other is fitted $\lambda$ in TableGames. If the heterogeneous model loses less to shirkers and shows a higher fitted $\lambda$, while matching accuracy in all-cooperative clusters, the claim holds: verifiable shared-evidence play teaches conditional prosociality that transfers. If not, the result tells us that retrieval-grounded training teaches coordination but not a transferable social disposition. The secondary weight-difference analysis then helps show whether any gain sits in separable parameters or is tied to retrieval skill.
References
- Mechanism Design Is Not Enough: Prosocial Agents for Cooperative AI
- Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questions
- ♫ MuSiQue: Multihop Questions via Single-hop Question Composition
- Before Reasoning Can Fail: Pre-Evidence Procedural Failures in Agentic RAG
- FrugalRAG: Less is More in RL Finetuning for Multi-hop Question Answering
- PRISMA: Reinforcement Learning Guided Two-Stage Policy Optimization in Multi-Agent Architecture for Open-Domain Multi-Hop Question Answering
- Learning Roles With Emergent Social Value Orientations
- Evaluating Cooperation in LLM Social Groups through Elected Leadership
- Adaptive Collaboration with Humans: Metacognitive Policy Optimization for Multi-Agent LLMs with Continual Learning
- Reasoning Vectors: Transferring Chain-of-Thought Capabilities via Task Arithmetic