Which Agent Earned It? Uncertainty-Aware Shapley Credit for Roles in LLM Pipelines

Multi-agent language-model systems split a hard task into roles, such as an analyst who plans, a coder who writes, and a tester who critiques, and they report large gains over a single model call. What they almost never report is which roles produce those gains. A role that adds 30% more tokens may contribute nothing on most problems, or it may matter only when another role is missing. The standard evidence is a leave-one-out ablation run once. LLM outputs are stochastic, and problems differ in which help they need, so a single ablation number mixes two kinds of uncertainty. The first is that a role truly helps on some problems and not others. The second is that repeating the same run gives a different answer.

We propose measuring each role's contribution exactly, as a Shapley value over role coalitions. Pipelines have only three to five roles, so every subset can actually be executed. Each role's credit is then summarized with a cloud model, a representation from risk engineering that describes a vague concept by three numbers: a center, a breadth that captures genuine variation, and a "thickness" that captures noise in that breadth. The goal is practical. We want to decide which roles a pipeline can drop, which it must keep, and which it should call only on some problems, and to spend fewer tokens at matched accuracy than the ablation practice used today.

Core Claim

Because role counts are small, exact role-level Shapley values are affordable in LLM pipelines. This removes the need for the heuristic, self-estimated credit used by prior work. The resulting credit is only trustworthy if between-problem heterogeneity is kept separate from run-to-run sampling noise. We claim three things. First, leave-one-out role rankings in published-style pipelines are unstable across seeds and misattribute credit between redundant roles, such as a tester and a reviewer that catch the same bugs. Second, a two-level uncertainty summary of Shapley credit supports pruning and routing decisions that beat both leave-one-out and single-number Shapley pruning on held-out problems. Third, LLM agents' own estimates of their contribution, the mechanism Shapley-Coop relies on, correlate only weakly with measured credit. The cloud model is the reporting and decision layer, not the source of the gain. We test directly whether it adds anything over a plain Gaussian random-effects summary, and we treat equivalence as a publishable and useful outcome.

Research Program

Pipelines and base solver. We re-implement three role decompositions from the literature:

  • the Analyst–Coder–Tester team of Self-collaboration, evaluated on HumanEval, MBPP and their edge-case variants;
  • the Thinker–Judge–Executor condition-mining loop of MACM, evaluated on Level 5 MATH;
  • the Generator–Reviewer–Inspector loop of ReChisel, evaluated on its 216-case Chisel suite.

In each pipeline, the base is a single call to the same model with the same task. The "players" are the added roles.

Removing a role. When a role is absent, its message slot passes through unchanged. For example, the coder sees the raw task instead of an analyst's plan, and a missing judge accepts every condition. This design choice mirrors the imputation problem in feature attribution. Masking a feature with background values creates unrealistic inputs and can distort attributions, as shown in Unfooling SHAP and the ICLR 2023 study. By analogy, removing a role could leave its neighbors seeing inputs they were never prompted for. We therefore run a sensitivity check with a second removal rule that re-prompts neighbors to expect no input.

Measuring coalition value. For a problem $p$ and a subset $S$ of roles, let $v_p(S)$ be the pass or accuracy rate of the pipeline restricted to $S$, averaged over $k$ independent seeds. The per-problem Shapley credit of role $i$ among $n$ roles is

$$\phi_i(p) = \sum_{S \subseteq N \setminus \{i\}} \frac{|S|!\,(n-|S|-1)!}{n!}\,\big[v_p(S\cup\{i\}) - v_p(S)\big],$$

where $N$ is the role set. With $n \le 5$, this is at most 32 coalitions per problem. For HumanEval with three roles and $k=5$, that is about 6,600 pipeline runs, which is affordable. The efficiency axiom (Holzinger et al.) guarantees that the credits sum exactly to the full pipeline's gain over the base call. Leave-one-out does not guarantee this.

Summarizing credit with a cloud. Cloud models, as used in tunneling and pipeline risk assessment (Wei Wang et al.; Han et al.), describe a concept by three parameters:

  • expectation $Ex$, the center;
  • entropy $En$, the breadth, meaning how far values range while still belonging to the concept;
  • hyper-entropy $He$, the dispersion of that breadth.

For each role, we set $Ex_i$ as the mean of $\phi_i(p)$ over problems and $En_i$ as their between-problem standard deviation. We set $He_i$ as the standard deviation of $En_i$ across bootstrap resamples of the $k$ seeds. This departs from the classical inverse cloud generator, which infers $He$ from higher moments of a single sample. We estimate $He$ directly because our design has an explicit seed level.

We also define three graded importance concepts, "negligible," "conditional," and "essential," as clouds over credit relative to the role's token cost. The forward cloud generator then gives each role a stochastic membership in each concept. This is the same device that places a monitoring group in the gray zone between "warning" and "danger" in Wei Wang et al..

Decision rule. A role is dropped if it is mostly "negligible." It is kept if it is mostly "essential." If it is mostly "conditional," which means low $Ex$ but large $En$, it is handed to a cheap router. The router is a small classifier on problem text that predicts whether $\phi_i(p)$ will be positive, and the role is called only when that prediction is positive.

Self-assessed credit. Following Shapley-Coop, we prompt agents to estimate each role's marginal contribution after each trajectory, and compare these estimates with the measured $\phi_i(p)$.

How the three directions divide the work. Multi-agent decomposition is the object of study. Shapley attribution is the measurement. The cloud model is the uncertainty representation and decision layer. The second use of game theory in Wei Wang et al. is different: it computes a Nash compromise between global AHP and entropy weights. It gives no per-instance attribution and involves no agents.

Baselines

The do-nothing reference. This is the full pipeline with every role always on, run with the same model, the same seeds, and the same problem splits as our method. It fixes the accuracy ceiling and the token cost that pruning must approach. For context, Self-collaboration reports 90.2% Pass@1 on HumanEval with GPT-4. MACM reports 76.73% on Level 5 MATH with GPT-4 Turbo, up from 54.68%. ReChisel reports 84.98% Pass@1 with Claude 3.5 Sonnet, up from 33.33%.

Our backbone will differ from those papers, so these figures only anchor plausibility. All comparisons are made within our own reruns. We report the base single call alongside the full pipeline so that the total gain being divided up is explicit.

The strongest existing practice: leave-one-out ablation. This is the ablation reported in the papers above. Each role is removed once and the accuracy drop is recorded. It is the true competitor because it is what practitioners use to justify a role. For a fair comparison, it gets the same $k$ seeds and the same compute budget. Its extra budget is spent on more seeds instead of more coalitions. Its pruning rule drops roles whose drop falls below a threshold, and that threshold is tuned on the same validation split as ours.

The single-number Shapley ablation. This baseline isolates the cloud layer. It uses the exact Shapley means $Ex_i$ with a bootstrap confidence interval, and it prunes by thresholding the interval's lower bound.

The two-level statistical ablation. This is a Gaussian random-effects model, where problem-level credit varies around a role mean and seed noise sits within each problem. Its decisions come from posterior probabilities instead of cloud memberships. It is the fairest test of whether cloud machinery adds anything.

The self-assessment baseline. This is Shapley-Coop-style LLM-estimated credit, used as a drop-in replacement for measured credit in the same decision rule.

What counts as beating the baselines. On held-out problems, our method must reach an accuracy within one point of the full pipeline at a significantly lower token count than leave-one-out pruning at matched accuracy. It must also show a lower seed-to-seed flip rate in keep-or-drop decisions. Beating the random-effects ablation is not required for the core claim. We report that comparison either way.

Why not just…?

Why not just use leave-one-out ablation with more seeds? More seeds fix the noise problem but not the redundancy problem. Suppose the tester and reviewer each catch the same bug class. Removing either one alone costs little, so leave-one-out rates both as dispensable. Dropping both could then be disastrous. This is the collinearity failure of leave-one-out discussed for feature attribution by Holzinger et al.. Our all-coalition runs directly measure the accuracy of the "drop both" configuration that leave-one-out would recommend, so the experiment will show whether this failure occurs in real pipelines. If no pipeline has redundant roles, leave-one-out with more seeds may suffice, and we will say so.

Why not just ask the agents who contributed, as Shapley-Coop does? Shapley-Coop has LLM agents estimate Shapley splits inside chain-of-thought. It reports alignment within 6% of human earned-value accounting for programmer roles in a ChatDev setting. However, that alignment is with human bookkeeping, not with counterfactual task success. The paper itself notes that exact computation was intractable over its action trajectories. LLM-MCA likewise uses an LLM critic to split team reward heuristically, without axiomatic guarantees. Our setting makes the exact counterfactual computable, so we can measure how far self-report deviates from it. If self-report tracks measured credit closely, it becomes the cheap option. That would be a real threat to our method's cost argument, and the rank-correlation experiment settles it.

Why not just use fuzzy scoring or a random-effects model instead of a cloud model? Fixed fuzzy memberships assign one crisp degree to each role and ignore seed noise. Wei Wang et al. report that fuzzy comprehensive evaluation produced distorted tier assignments where the cloud model did not, but that was in tunneling, not LLM pipelines. The Gaussian random-effects model is the more serious competitor. It also separates heterogeneity from noise, and it may perform just as well. We run it as a full baseline. If it matches, the contribution narrows to the two-level exact Shapley design plus the routing of "conditional" roles, and the cloud layer serves as an interpretable graded interface rather than a performance gain.

Proposed Experiments and Decisive Checks

We compute exact role Shapley values with $k=5$ seeds on three benchmarks: HumanEval together with HumanEval-ET, a 500-problem Level 5 MATH subset, and the Chisel suite. Each is split 60/20/20 into calibration, validation and test sets.

Check 1: ranking stability. We resample seeds and count how often the leave-one-out and Shapley role rankings flip. A flip rate above roughly 20% for leave-one-out would show that single-run ablations in the literature are not reliable evidence.

Check 2: redundancy. For each pair of roles, we compare the accuracy of dropping both, as leave-one-out would recommend, against their joint Shapley credit and a pairwise interaction term.

Check 3: pruning under a token budget. We trace accuracy-versus-tokens curves on the test split for the full pipeline, leave-one-out pruning, single-number Shapley pruning, random-effects pruning, self-assessed pruning, and cloud pruning with routing. We report the token saving at matched accuracy.

Check 4: routing. We test whether roles rated "conditional," with high $En$, can be predicted per problem by a cheap router well enough to beat both always-on and always-off. For example, the tester might matter mostly on edge-case problems in HumanEval-ET.

Check 5: self-assessed credit. We measure the Spearman correlation between Shapley-Coop-style self-assessed credit and measured $\phi_i(p)$.

Check 6: robustness. We repeat the main analysis under the alternative role-removal rule and with a smaller open model. The smaller model tests whether credit shifts to the judge and tester roles, since smaller models are reported to break role boundaries (Dong et al.; Lei et al.).

What would sink the proposal. The proposal fails if leave-one-out rankings are stable, pairwise redundancy is absent, and self-assessed credit tracks measured credit. In that case, cheap methods suffice and exact coalition runs are not worth their cost.