TL;DR
This paper investigates whether interpreter persistence—the ability of an LLM agent's coding environment to retain variables across turns—is a fundamental property absorbed during training or merely an inference-time convenience. Through a controlled 2×2 study, the authors demonstrate that mismatching training and deployment runtime semantics doesn't significantly impact final solution quality, but it severely penalizes efficiency and stability. Misaligned agents either waste massive token budgets re-deriving state (an "amnesia tax") or collapse into destructive error loops.
What Problem Does This Paper Address?
Many state-of-the-art LLM agent frameworks (like CodeAct) allow agents to interact with a persistent Python interpreter, meaning variables and data structures defined in Turn 1 remain accessible in Turn 3. However, the synthetic datasets used to fine-tune these models rarely make this environmental assumption explicit.
This creates a blind spot: Do LLM agents actually learn and rely on the persistence of their runtime environment during post-training, or is persistence just an execution scaffold? Understanding this alignment matters because developers frequently train models in one environment setup and deploy them in another, unaware of how subtle runtime mismatches might alter agent behavior.
Methods
The authors design a rigorous 2×2 cross-evaluation study that isolates interpreter persistence as a training-time variable. They hold the base model (Qwen3-8B), prompts, tools, and task instances completely fixed, while manipulating two dimensions:
- Training Semantics: Fine-tuning on trajectories generated with a persistent interpreter vs. a stateless (reset each turn) interpreter.
- Runtime Semantics: Evaluating the fine-tuned models in a persistent vs. stateless deployment environment.
To ensure fair comparison, the runtime explicitly exposes metadata about active global variables to the agent in both setups, isolating the actual execution mechanics from pure observability.
Datasets
The authors introduce OPAQUE KNAPSACK, a procedurally generated, partially observable optimization benchmark. Unlike standard tool-use tasks that can be solved with a single long script, this environment hides item weights, values, and class constraints behind a budgeted tool API (list_items, inspect, take_item).
Because information acquisition is budget-limited and feedback-driven, the task is strictly "non-collapsible"—agents are forced to iteratively gather information, track state, and revise plans over multiple turns. The dataset consists of 1,000 Easy instances for training and 100 Hard instances for testing difficulty generalization.
Key Insights and Ideas
- Persistence is a Learned Prior, Not Just a Prompt: When a persistent-trained model is forced into a stateless runtime, it doesn't adapt dynamically—even when explicitly shown stateless few-shot examples. It continues to guess and reference old variables, proving that execution semantics are absorbed as a deep behavioral prior during fine-tuning.
- The "Amnesia Tax": Stateless-trained models carry an ingrained habit of text-based state reconstruction. When deployed in a persistent runtime, they completely fail to exploit the interpreter's memory. Instead, they redundantly re-import libraries and re-define variables on every single step, incurring a massive token overhead.
- Productive Externalization vs. Destructive Thrashing: While both a stateless-aligned agent and a mismatched persistent-to-stateless agent consume high token volumes, they do so for opposite reasons. The stateless agent uses tokens productively to rebuild state via text. The mismatched persistent agent wastes its token budget on cascading recovery loops trying to fix
NameErrorexceptions.
Key Results
- Solution Quality is Unaffected: Surprisingly, across all four train-runtime combinations, there was no statistically significant difference in final solution optimality. Interpreter persistence shapes how agents reach solutions, not whether they do.
- Massive Token Savings: On Hard tasks, the aligned persistent configuration (Persistent $\rightarrow$ Persistent) is highly efficient, requiring roughly 3.5× fewer tokens than the aligned stateless configuration (67,898 vs. 18,612 tokens).
- High Error Rates in Mismatched Deployments: Deploying a persistent-trained model in a stateless environment triggers missing-variable errors (
NameError) in ~80% of episodes, frequently causing the agent to enter cascading execution failures that consume its entire turn budget.
Limitations and Future Work
The authors note several limitations in their current study:
- Evaluation Power: The sample size ($n = 100$ tasks per split) is sufficient to expose massive behavioral and efficiency shifts, but underpowered to detect subtle differences in absolute solution optimality.
- Token-Budget Confounds: Matching training sets by episode count meant that the stateless-trained model was exposed to substantially more raw text during training (~3.5×). Future work should ablate token-matched training budgets.
- Scope of Generalization: The findings are currently bounded by a single task family (Opaque Knapsack) and a single base model (Qwen3-8B). Testing across varied agent architectures and diverse benchmarks remains an important next step.
Summary updated