One Run Is Not Enough: Reminding a small Gemma 4 coding agent of its last finding made no measurable difference in a paired test

Patrick PassosarXiv 2026 · arXiv

This paper, authored by Patrick Passos and published in late 2026, serves as a sobering reality check for the AI agent community. It investigates whether a simple "memory" fix can help a small coding agent (Gemma 4 31B) overcome context window limitations, while simultaneously exposing the high level of statistical noise inherent in LLM benchmarking.

TL;DR

Adding a simple "recap" mechanism to remind a coding agent of its last finding after context compaction resulted in a net-zero performance gain across 98 tasks. The study highlights a critical "noise floor" in agent evaluation: nearly 20% of tasks changed their outcome simply by running the exact same model twice, suggesting that many reported gains in the field may be statistical mirages.

What Problem Does This Paper Address?

Coding agents often operate over long trajectories that exceed their context window. When the window fills up, the "harness" (the software managing the agent) must compact or truncate the conversation. This causes the agent to "forget" earlier discoveries, leading to redundant work or failure.

The authors tested a straightforward intervention: automatic re-injection. If the agent’s explorer sub-agent found something important right before compaction, the harness would automatically put that finding back into the prompt. The goal was to see if this "reminder" would help a small model (Gemma 4 31B) resolve more real-world repository issues in the SWE-bench family.

Methods

The researchers used a rigorous, pre-registered paired test design:

  • The Model: Gemma 4 31B (quantized to 4-bit), using a "thinking budget" of 4096 tokens.
  • The "Recap" Arm: After context compaction (triggered at 14,336 tokens), the harness injected the explorer's latest finding (up to 15 lines) and any agent-written notes into every subsequent request.
  • A/A Calibration: Crucially, the authors ran a "noise floor" test, comparing the baseline model against itself (Run 1 vs. Run 2) with no changes.
  • Validity Funnel: They filtered the initial 129 tasks down to 105 "valid" tasks where the environment and "gold" reference patches were confirmed to work, ensuring failures were due to the agent, not the setup.

Datasets

The study utilized the SWE-bench family, specifically focusing on 129 public development tasks from popular Python repositories:

  • fastapi (67 tasks)
  • rich (48 tasks)
  • requests (13 tasks)
  • httpx (1 task)

Key Insights and Ideas

  • The Mirage of Success: In the first 48 tasks, the memory arm seemed to win by +8.3 points. However, when they extended the study by another 50 tasks (as per their pre-registration), the result reversed to -10.0 points. This "sign flip" proves that small-sample successes can be entirely random.
  • The "Noise Floor" is High: 19% of tasks flipped their outcome (from solved to unsolved or vice-versa) between two identical runs of the no-memory baseline. This means that for a task set of this size, any "improvement" smaller than 12-18 points is likely just run-to-run noise.
  • Agents are "Lazy" Note-Takers: Even when provided with a tool to write working notes, the Gemma 4 model almost never used it (writing notes in only about 12% of runs). Memory systems that rely on agent cooperation may be doomed for smaller models.
  • The Context Tax: Memory isn't free. Injecting "reminders" consumes context space, which actually triggered more frequent compactions in the memory arm, potentially creating a "vicious cycle" of forgetting.

Key Results

  • Primary Outcome: Across 98 paired tasks, the memory arm solved 47 and the no-memory arm solved 48. This is a statistically null result ($p = 1.0$).
  • Secondary Metrics: There was no significant difference in "repeated actions" (the agent doing the same thing twice because it forgot it already did it).
  • A/A Discordance: 9 out of 48 tasks changed outcome with no treatment at all, establishing a paired standard error of ~0.063.

Limitations and Future Work

  • Model Size: The results are specific to a 31B model. Larger models (like GPT-4o or Claude 3.5) might be better at utilizing injected memory or writing their own notes.
  • Simple Mechanism: The study did not test more complex memory architectures like RAG (Retrieval-Augmented Generation) or long-term vector databases.
  • Future Direction: The authors suggest moving toward a "recap" system that builds summaries from tool history automatically, removing the need for the agent to "choose" to remember. They also recommend implementing "guards" to stop agents from entering infinite loops (e.g., calling submit_patch hundreds of times with the same code).

Ultimately, the paper serves as a warning: One run is not enough. If you aren't measuring your noise floor, you aren't measuring your progress.

Summary updated

Abstract

When a coding agent runs out of room, its earlier findings are cut and it forgets them. We tested a simple fix: automatically putting the agent's last finding back in front of it. We fixed the rules in advance and ran 98 paired coding tasks with and without this reminder, using Gemma 4 31B (QAT, 4-bit) in the Gemma 4 Developer Agent harness. The reminder made no measurable difference: across all 98 tasks, 47 were solved with it and 48 without (-1.0 points, 95% interval [-8.3, +6.3], exact McNemar p=1.0). The primary outcome (a documented deviation from our preregistration), the first 48 pairs, was 29/48 with memory against 25/48 without (+8.3 points, [-2.1, +18.3], p=0.22); the extension of 50 tasks, decided after that result, pointed the other way (18/50 against 23/50, -10.0 points, p=0.125), so the sign did not replicate. With 19% of tasks flipping between two identical no-memory runs, the combined interval excludes harms larger than about 8 points and gains larger than about 6, and the design had 80% power only for effects of about +-12 points or more. The model almost never wrote notes (2 of 16 and 4 of 32 runs), and memory cost context: the first request is larger and something was injected at only 47% to 65% of compactions. The registered secondary, repeated actions per compaction, was also null. Two directional hypotheses found on Modal did not replicate on Kaggle. We also report an A/A calibration and a validity funnel (129 tasks, 105 valid). Together these results show that single-run memory claims on task sets of this size sit inside run-to-run noise. The treatment is a harness mechanism and does not test agent-written memory.
BibTeX
@misc{passos2026one,
  author = {Patrick Passos},
  title = {{One Run Is Not Enough: Reminding a small Gemma 4 coding agent of its last finding made no measurable difference in a paired test}},
  year = {2026},
  eprint = {2611.19359},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.19359},
}
Figure