TL;DR
Coding agents often perform redundant, "over-defensive" actions—like rerunning tests that have already passed—because standard training methods can't distinguish between doing "just enough" and doing "too much" if both lead to success. This paper identifies that these inefficiencies begin at silent forks, decision points where multiple actions lead to the same successful outcome, and introduces ForkDPO to provide a targeted learning signal at these exact moments. By training on these forks, the authors reduced a 4B model's redundant behavior and cut median agent runtime by 37% on unseen tasks.
What Problem Does This Paper Address?
When we train LLMs to act as coding agents, we typically use outcome rewards: did the agent successfully fix the bug or pass the repository's test suite? While effective for teaching success, this approach is "blind" to efficiency. If an agent can either "submit the fix" or "rerun the entire test suite for the fifth time," and both lead to a successful outcome, an outcome-based reward treats them as identical.
This leads to over-defense: agents that are technically successful but incredibly slow and computationally expensive. Current reinforcement learning (RL) methods often try to solve this by penalizing long trajectories or using process-based rewards. However, the authors find that over-defense is particularly slippery because it starts at what they call silent forks. At these points, the probability of success is already settled, so standard credit assignment methods (which look for actions that change the chance of success) see no signal to learn from.
Methods
The authors propose a two-step approach to diagnose and fix over-defense:
- Behavioral Checker: Instead of just looking at the final outcome, they use a rule-based checker to replay agent trajectories. This checker identifies "proportionate" behavior (doing what is required) versus "disproportionate" behavior (redundant tests, extra checks, etc.).
- ForkDPO (Fork-level Direct Preference Optimization):
- Standard DPO often compares a full "winning" trajectory to a "losing" one, or samples pairs at random turns.
- ForkDPO specifically targets the silent fork. It identifies the first turn where a successful agent starts over-defending.
- It then resamples the model at that exact prefix to find a "boundary action"—the minimum necessary step to proceed (e.g., submitting if tests passed).
- The model is then trained to prefer the proportionate boundary action over its original over-defensive action.
Datasets
The study uses a newly synthesized collection of 368 tasks derived from 34 open-source Python and Go repositories.
- Training: 318 tasks from 30 repositories.
- Testing: 48 tasks from 4 completely unseen repositories to ensure the model learns general behavior rather than just memorizing specific repo structures.
- The tasks are categorized into "explicit budget" (where the prompt specifies exactly what to run) and "implicit budget" (where the agent must infer the necessary checks).
Key Insights and Ideas
- Frontier Models are Paranoid: Even state-of-the-art models like GPT-4o and Claude 3.5 Opus suffer from extreme over-defense. When budgets are implicit, these models take disproportionate actions in 81–91% of decisive turns.
- The "Silent" Nature of Inefficiency: Because over-defensive actions (like rerunning a test) don't change the repository state or the ultimate success probability, they create a "vanishing gradient" for traditional RL. The signal literally disappears because the outcome is already "settled."
- Timing Matters: Redundant behavior usually starts very early. The authors found that the median "silent fork" occurs after only two actions.
- Trajectory Penalties Fail: Methods like GRPO (Group Relative Policy Optimization) struggle here because as tasks get longer, the probability of an agent being perfectly proportionate throughout the entire run becomes vanishingly small. If no one in a group is efficient, there's no "good" example to learn from.
Key Results
The authors tested ForkDPO on a 4B parameter model (Qwen3.5-4B) and compared it against standard DPO and GRPO:
- Reduced Redundancy: ForkDPO lowered the disproportionate behavior rate from 70.3% to 58.9% on unseen repositories.
- Significant Speedup: By eliminating redundant test runs, the median wall-clock runtime for the agent dropped from 143 seconds to 90 seconds—a 37% improvement.
- Better than Baselines: Standard DPO at random turns only reduced disproportionate behavior to 67.7%, proving that where you place the learning signal is more important than just having the signal.
- Maintained Success: Crucially, the model did not become "under-defensive." By pairing over-defense forks with "skipped-check" forks (where the agent tries to submit too early), ForkDPO taught the model the "Goldilocks" zone of doing exactly what is required.
Limitations and Future Work
- Rule-Based Limitations: The behavioral checker relies on hand-written rules for what constitutes "over-defense." While these rules showed high agreement with human annotators (0.93 precision), they may not capture more subtle forms of inefficiency in complex, multi-tool environments.
- Model Scale: Due to compute constraints, the authors primarily trained a 4B model. While they diagnosed frontier models, they did not apply ForkDPO to them.
- Sequential Over-Defense: The current method targets single turns where a model diverges. It may not catch "death by a thousand cuts" scenarios where a sequence of individually reasonable actions adds up to an inefficient whole.
- Future Direction: The authors suggest this "behavioral divergence" signal could be applied beyond coding to any agentic task where efficiency and cost-to-result are critical, such as web browsing or tool-use in general reasoning.
Summary updated