TL;DR
AI coding agents often solve complex software tasks much earlier in their execution than previously thought, but they struggle to recognize their own success. This "termination lag"—the gap between the first correct patch and the final submission—accounts for 35% to 60% of an agent's total work, incurring significant unnecessary costs and occasionally leading the agent to break its own working solution.
What Problem Does This Paper Address?
Current evaluations of autonomous software engineering agents (like those on the SWE-bench leaderboard) typically only look at the final patch submitted by the agent. This "black box" approach hides a critical inefficiency: agents often continue to work long after they have already generated a correct fix.
This behavior, which the authors call Termination Lag (TL), presents three major issues:
- Cost and Latency: Every extra step involves expensive LLM calls over a growing context window.
- Evaluation Conflation: It mixes two distinct skills—the ability to generate a fix and the ability to verify that the fix is complete.
- Coherence Collapse: In some cases, an agent that has already solved the task continues to edit the code until it eventually breaks the working solution, resulting in a failed submission.
Methods
The researchers conducted an empirical analysis of agent trajectories across five different LLMs (including Qwen3-Coder, Laguna-XS.2, Claude Haiku 5.5, and GPT-6 Luna) and two agent frameworks (OpenHands and mini-SWE-agent).
To "peek inside" the agent's process without influencing it, they:
- Recorded the state of the repository patch after every single step the agent took.
- Evaluated every distinct patch offline against the official benchmark tests (which the agent never sees).
- Defined two key metrics: Time to First Correct Patch (TTFCP) and Termination Lag (TL).
By comparing when the first correct patch appeared versus when the agent actually stopped, they could quantify exactly how much effort was "wasted."
Datasets
The study utilized three major benchmarks to ensure the findings generalized across languages and task difficulties:
- SWE-bench Verified: 500 curated Python tasks from real-world GitHub issues.
- SWE-bench Multilingual: 300 tasks spanning nine different programming languages.
- SWE-bench Pro: A set of harder, longer-horizon tasks requiring more sustained execution.
Key Insights and Ideas
- The "First-Edit" Success: Surprisingly, in 38–82% of successful runs, the agent's very first source-code edit was already correct.
- Verification is the Main Time Sink: Agents aren't necessarily "over-coding." Table 3 shows that 48–63% of the steps taken during the termination lag are spent on verification (running tests, writing check scripts) and 15–29% on exploration (searching the repo).
- The Risk of Overthinking: Continued activity isn't just neutral; it's risky. In 1–8% of cases, the agent "snatched defeat from the jaws of victory" by overwriting a correct patch with an incorrect one before terminating.
- Context Bloat: Because agents operate over a conversation history, the termination lag causes the input context to grow by 28% to 78% (see Table 1), further driving up the computational cost of the final steps.
Key Results
- Substantial Lag: In successful runs, the median Normalized Termination Lag (NTL)—the fraction of the run occurring after the fix—was 35–37% for Qwen3-Coder and a staggering 54–60% for Laguna-XS.2.
- Efficiency Gains: An "oracle" that could stop the agent the moment it found a correct fix would reduce total steps by roughly a third to more than half across all models tested.
- Consistency Across Models: Whether using open-source models like Qwen or top-tier closed models like GPT-6 Luna, the pattern remained: agents solve early but stop late.
Limitations and Future Work
The authors acknowledge several limitations:
- Single-Run Statistics: Each task was run only once per setting; results might vary with multiple seeds.
- Step vs. Cost: While they measure "steps," this is only a proxy for actual dollar cost or FLOPs, though the context growth data suggests the cost per step increases over time.
- Heuristic Categorization: The categorization of agent actions (Verify vs. Explore) is based on rules applied to tool usage, which may not capture the full nuance of the agent's "intent."
Future Work: The clear next step is developing better stopping rules. If we can build agents that better recognize when a task is "done" using only the evidence available to them (without seeing the hidden benchmark tests), we can drastically reduce the cost and increase the reliability of autonomous software engineering.
Summary updated