Retrieval-augmented question answering systems are usually pipelines: one model rewrites the question, another picks which retrieved documents to keep, and a third writes the answer. When such a pipeline is trained end to end with reinforcement learning, every stage gets the same reward, which is the quality of the final answer. That shared reward tells the answer writer it failed even when the real mistake was the document picker throwing away the one passage that mattered. We propose to replace this shared signal with a blame-aware one. When an answer is wrong, a large language model acting as a grader (an "LLM judge") reads the intermediate outputs and labels each stage in one of three ways. The stage originated the error, or it inherited a bad input and passed it along, or it repaired a bad input from upstream. Each stage is then rewarded or penalized only for what it actually did.
The idea in two sentences: shared end-of-pipeline rewards punish downstream stages for upstream mistakes, and the per-step grading methods that exist today make the same error in a different form, because they score each step as if its input were correct. Separating "originated" from "inherited" from "repaired" gives each stage a reward it can actually influence, and applying the judge only to failed runs keeps the cost proportional to the failure rate rather than the number of steps.
Core Claim
Our claim is that what helps multi-stage training is not denser feedback in general but feedback that respects which stage caused the problem. Concretely, for stage $k$ on a trajectory with final-answer score $F$ (token F1), the reward is
$$ r_k = F + \lambda \, a_k, \qquad a_k \in \{-1, 0, +1\}, $$
where $a_k = -1$ if the judge says stage $k$ originated the failure, $0$ if it was clean or only inherited a bad input, $+1$ if it repaired an upstream error, and $\lambda$ is a weight tuned on held-out data. The judge is called only when $F$ falls below a threshold. On successful runs every stage simply receives $F$.
The evidence motivating this comes from two papers that have not yet been connected. AgentEval builds a step-level diagnostic. A GPT-4o judge scores each node of an agent workflow against its immediate upstream input. It finds that 63% of step failures propagate from upstream rather than being local mistakes. It also reports that outcome-only evaluation reaches failure-detection recall of 0.41 against 0.89 for dependency-aware evaluation, and 72% root-cause accuracy against a human ceiling of 81%. AgentEval stops at diagnosis and never trains anything.
MMOA-RAG does train a three-stage pipeline: a query rewriter, a document selector, and an answer generator. The three stages share a Llama-3-8B-Instruct backbone, and training uses multi-agent proximal policy optimization, a policy-gradient method for several cooperating agents. Its single shared terminal F1 reward beats modular baselines by +1.80 F1 on HotpotQA, +1.89 on 2WikiMultihopQA and +2.67 on AmbigQA.
The tension between them is the opportunity. If most failures originate upstream, MMOA-RAG's shared reward is assigning blame to the wrong stage most of the time. We propose to turn AgentEval's diagnosis into MMOA-RAG's training signal, with one structural change: a third label for repair.
The repair label comes from Beyond the First Error. That paper had an o1 judge distinguish "error propagation" from "error cessation" inside a single model's math reasoning chain, and showed the distinction matters for training a step-level grader. We carry that distinction across model boundaries. A generator that answers correctly despite a degraded document set is doing something we want to reinforce, not ignore.
Relation to Turn-Level Credit Assignment
The closest existing work is a family of methods that already give intermediate credit in multi-step agents.
MT-PPO adds a per-turn LLM-judge reward plus format checks to a single search agent. It reports 45.3% on HotpotQA against GRPO's 41.6%, and notes that per-turn judging is costly and invites reward hacking.
iStar avoids per-step judging by learning an implicit step reward from trajectory preferences. HCAPO uses the policy itself as a hindsight critic. ITPO derives turn-level implicit rewards for dialogue.
All four assume a single policy acting turn by turn. All four score each turn's quality rather than its causal role. Our proposal differs on both counts. The unit of credit is a pipeline stage with its own input and output type. The judge's job is attribution along a dependency chain, not quality scoring.
Research Program
The work has three parts.
Attribution judge. We first build the judge and validate its attributions against a cheap source of ground truth. HotpotQA and 2WikiMultihopQA annotate supporting facts, which lets us run counterfactual repairs. If injecting the gold supporting passages into the selector's output flips a wrong answer to right, the selector (or rewriter) is the true origin. If the answer stays wrong even with gold passages, the generator originated it. If the answer was right despite missing gold passages, the generator repaired.
These counterfactual labels let us measure the judge's attribution accuracy directly, rather than trusting human agreement alone. We expect, but must verify, that the AgentEval-style restriction helps. That restriction means showing the judge only each stage's immediate input and output and asking for a categorical label rather than a scalar.
Known judge failures are a real risk to the design. The risks include shortcut cues (The Silent Judge), position and sentiment bias (Justice or Prejudice?), and overconfidence (Overconfidence in LLM-as-a-Judge). Categorical labels on short inputs are meant to limit verbosity effects, and we will measure bias by permuting document order and padding outputs.
Training. Second, we train the MMOA-RAG pipeline with blame-aware rewards, keeping its backbone, algorithm and data. The terminal F1 term stays dominant. This guards against stages learning to please the judge at the expense of the answer.
Cost-matched credit assignment. Third, we test whether the judge earns its cost against a judge-free alternative: counterfactual credit computed by actually substituting gold or perturbed intermediate outputs and re-running downstream stages. Judge calls are counted and charged against a fixed compute budget for every method.
Baselines
Untrained pipeline. The naive reference is the untrained pipeline (same backbone, prompted only). This tells us how much any training buys.
Modular supervised fine-tuning. Each stage is fine-tuned on its own target: the rewriter on rewrites, the selector on supporting-fact labels, the generator on gold contexts. We reproduce this in MMOA-RAG's setup on the same data. It represents the "optimize parts in isolation" view that Searching for Best Practices in RAG and MMOA-RAG argue against. MMOA-RAG reports beating it.
MMOA-RAG itself. The strongest existing method on this exact question is MMOA-RAG, with its shared terminal F1 reward under MAPPO. The file reports its gains as +1.80, +1.89 and +2.67 F1 over the best baselines on HotpotQA, 2WikiMultihopQA and AmbigQA. No absolute F1 for MMOA-RAG was found on Lacuna, so we will reproduce it ourselves. We will use the same backbone, the same number of environment interactions, and the same hyperparameter search budget, and give it extra rollouts equal in compute to our judge calls. Beating it means a higher final F1 on all three datasets at matched compute, with the gap exceeding seed variance over at least three seeds. Reaching MMOA-RAG's final F1 in clearly fewer samples would also count.
MT-PPO-style judge reward. We also adapt MT-PPO's per-turn judge reward to the three stages, scoring each stage's quality independently. Its reported 45.3% on HotpotQA comes from a single-agent search setting and is not directly comparable, so we run our own adaptation.
Flat-judge ablation. This is the key ablation and isolates our ingredient. The same judge, prompts and call budget are used, but the judge gives each stage a good or bad quality label with no originated, inherited or repaired distinction.
Two-label ablation. A second ablation drops only the repair label, testing whether rewarding downstream recovery matters.
Oracle reward. As an upper reference on HotpotQA, we reward the selector with supporting-fact recall. This is the rule-based oracle.
Why not just…?
Why not just keep the shared terminal reward and train longer? MMOA-RAG shows a shared reward works, so more samples might close any gap. The counterargument is that AgentEval's 0.41 versus 0.89 recall result suggests terminal signals hide most failures. In single-agent settings, MT-PPO reports that terminal-reward PPO and GRPO become unstable on multi-hop QA. We settle this empirically by giving MMOA-RAG our judge budget converted into extra rollouts and plotting F1 against compute. If the curves meet, our method buys sample efficiency only, and we will report it that way.
Why not just have the judge score every stage independently? This is the flat ablation, and it is the most natural reviewer suggestion. AgentEval's own ablation found flat evaluation misattributes downstream steps that faithfully processed bad inputs. Beyond the First Error found that a "first-error" assumption penalizes valid self-correction. Neither result was obtained in a training loop, though. If the flat judge matches the blame-aware judge in final F1, the three-way distinction is unnecessary, and the paper's claim fails.
Why not just use rule-based intermediate rewards like supporting-fact recall? On HotpotQA this is a real threat, which is why it is our oracle baseline. MTU-Bench's decay-weighted turn metric shows rule-based tracking works when gold slots exist (MTU-Bench). However, gold intermediate labels are absent for the rewriter on all datasets and for AmbigQA's ambiguous questions. Rule-based labels also cannot detect repair. We expect the oracle to be competitive on HotpotQA and our method to win where no gold intermediates exist. A loss on AmbigQA would undercut the method's main selling point.
Why not just compute credit by counterfactual re-execution instead of a judge? Substituting outputs and re-running downstream stages gives causal credit with no judge bias. Beyond the First Error found a judge more label-efficient than Monte Carlo rollouts for step labels, but that was within a single model's reasoning. We run counterfactual credit at matched compute. If it wins, the right conclusion is that the judge should be used only to propose which counterfactual to run, and we will say so.
Proposed Experiments
All experiments use HotpotQA, 2WikiMultihopQA and AmbigQA with MMOA-RAG's backbone, retriever and splits. Metrics are answer F1 and exact match, plus judge calls per training step and attribution accuracy against counterfactual labels.
Attribution validity. The first decisive check is whether the judge's originated, inherited and repaired labels agree with the gold-passage counterfactual labels well above the rate obtained by always blaming the final stage. AgentEval's 72% root-cause accuracy is our reference point. If agreement is low, training on these labels is pointless and the project stops there.
Ablation tracking. The second check is whether the blame-aware reward beats the flat-judge and two-label ablations at identical judge budgets. This isolates the contribution.
Matched-compute comparison. The third check is a compute-matched comparison against MMOA-RAG and counterfactual-rollout credit, with learning curves over environment samples.
Cost. The fraction of trajectories that trigger the judge is reported throughout. Judging only failures is meant to make cost fall as the pipeline improves.
Behavioral diagnostics. We log, per stage, how often each stage is the originator over training. We expect blame to shift away from the generator as the selector improves. We also run the bias probes (document-order permutation and output padding) to check that gains do not come from exploiting the judge.
References
- AgentEval: DAG-Structured Step-Level Evaluation for Agentic Workflows with Error Propagation Tracking
- Improving Retrieval-Augmented Generation through Multi-Agent Reinforcement Learning
- Beyond the First Error: Process Reward Models for Reflective Mathematical Reasoning
- Reinforcing Multi-Turn Reasoning in LLM Agents via Turn-Level Credit Assignment
- Agentic Reinforcement Learning with Implicit Step Rewards
- Hindsight Credit Assignment for Long-Horizon LLM Agents
- Implicit Turn-Wise Policy Optimization for Proactive User-LLM Interaction
- The Silent Judge: Unacknowledged Shortcut Bias in LLM-as-a-Judge
- Justice or Prejudice? Quantifying Biases in LLM-as-a-Judge
- Overconfidence in LLM-as-a-Judge: Diagnosis and Confidence-Driven Solution
- Searching for Best Practices in Retrieval-Augmented Generation
- MTU-Bench: A Multi-granularity Tool-Use Benchmark for Large Language Models