MarkovSim: an LLM Agent Harness with Bounded Context for Long-Horizon Computational Simulation

Yeui-Lung Lei, Wen-Bo Chen, Pan ZhangarXiv 2026 · arXiv

TL;DR

MarkovSim is a novel LLM agent harness designed to automate complex, long-horizon engineering simulations without succumbing to "context window" exhaustion. By utilizing a finite-state machine and a structured memory system to periodically reset the agent's focus, it maintains a bounded context regardless of task length, successfully completing 89% of tested finite element analysis (FEA) benchmarks.

What Problem Does This Paper Address?

Computational simulations, such as Finite Element Analysis (FEA) and Computational Fluid Dynamics (CFD), are essential for modern engineering but remain a significant bottleneck because they require scarce expert time. While Large Language Models (LLMs) show promise in automating these workflows, they face a fundamental technical hurdle: long-horizon reasoning.

A typical simulation involves dozens of interdependent steps—from geometry creation to results extraction—often requiring extensive trial and error. As an LLM agent works through these steps, its conversation history grows. Eventually, this history exceeds the LLM’s "context window," leading to two failure modes: the agent either stops working entirely or its performance degrades (the "lost in the middle" effect), causing it to forget constraints or hallucinate model parameters. MarkovSim aims to solve this by decoupling the agent's reasoning from the total length of the workflow history.

Methods

The authors introduce a context management scheme inspired by the Markov property, where the next state of a system depends only on the current state, not the entire history.

Key architectural features include:

  • Agentic Workflow: The process is divided into three specialized phases: Task Formulation (clarifying requirements with a human), Planning (decomposing the task into a hierarchical plan), and Execution (building and verifying the model step-by-step).
  • Finite-State Machine (FSM) Control: The Execution Agent moves through states like GENERATION, VERIFICATION, and REFINEMENT. Crucially, at designated FSM boundaries, the agent’s context is reset.
  • Structured Memory: Instead of relying on a chat log, the system writes the workflow record to a folder tree on disk. When a context reset occurs, the harness reconstructs a "Markov workflow context" for the agent. This includes the current step plan and a Model Tree—a compact JSON representation of the simulation model built so far.
  • Two-Layer Verification: As shown in Figure 4, every step must pass a Runtime layer (does the code execute?) and a Physical layer (does the resulting model represent the intended physics?). The physical check uses both quantitative data and visual renders of the model.
Schematics of the Execution phase and the two layers of verification

Datasets

MarkovSim was evaluated on six benchmark problems spanning four analysis types (static stress, thermo-mechanical, linear buckling, and free vibration):

  • NAFEMS Benchmarks: LE1 (Elliptic membrane), LE3 (Hemispherical shell), and T1 (Membrane with a hot-spot).
  • Abaqus Benchmarks: LBR (Lateral buckling of an L-bracket).
  • FEABench: FEA-12681G (Tapered cantilever) and FEA-75611 (Free vibration of a deep beam).

The system was connected to the commercial FEA software Abaqus for all tests.

Key Insights and Ideas

  • The Sawtooth Context: By resetting the context at each step, the token count follows a "sawtooth" pattern. It grows during a specific task but collapses back to a baseline (approx. 11,000 tokens) once the step is committed. This proves that an agent can run for over 1,000 calls without ever hitting a context limit.
  • Context Isolation: The authors use a dedicated "documentation subagent" to search software manuals. This prevents the messy, token-heavy process of reading documentation from contaminating the main Execution Agent's reasoning space.
  • Global Refinement: If a fault is discovered in a previously "committed" step, the system can perform a global refinement. It doesn't just move backward; it treats the repair as a new attempt in the current step, undoing and recreating artifacts in the model database to maintain consistency.
  • Intent vs. Implementation: Interestingly, all 27 human interventions required during testing concerned "engineering intent" (e.g., clarifying load directions) rather than software syntax. This suggests the agent is highly proficient at the "coding" aspect of simulation but still requires high-level guidance.

Key Results

  • High Success Rate: MarkovSim passed 16 out of 18 runs (89%).
  • Bounded Context: The peak context remained below 55,000 tokens across all problems, regardless of whether the run took 500 or 1,100 LLM calls.
  • Accuracy: Successful runs produced results within ±10% of the published reference quantities, which is considered sound for engineering applications where different solvers and mesh densities naturally cause slight variations.
  • Self-Correction: The system successfully diagnosed and repaired 7 out of 8 "global" faults that had initially passed local verification.

Limitations and Future Work

While successful, the authors note several areas for improvement:

  • Domain Specificity: The current instantiation is specific to FEA and Abaqus. The framework needs to be tested on CFD and other simulation methods to prove its generalizability.
  • Verification Sensitivity: The system is only as good as its verification checks. In one failed run, a "sign flip" in the stress results passed both the agent's checks and the human proxy's review, leading to an undetected error.
  • Human Emulation: The "engineer in the loop" was emulated by a high-end LLM (Claude Opus). Future studies should validate the system's collaboration with practicing human engineers.
  • Comparative Baselines: The paper focuses on demonstrating the MarkovSim architecture; future work is needed to compare it directly against other context management strategies like simple truncation or summarization.

Summary updated

Abstract

Recent studies have applied large language model (LLM) agent systems to computational simulation workflows. Such workflows are long-horizon, so the information generated can exceed what an LLM can reason over at once, its context window. Context management is therefore essential, yet existing systems have either neglected it or treated it as a supporting mechanism rather than a subject of study. This paper presents MarkovSim, an LLM agent harness for long-horizon computational simulation, whose core is a context management scheme built into the workflow. The workflow record is written to a memory system with a predefined schema, and at workflow-defined boundaries, namely the state transitions of a finite-state machine a compact representation of the solution progress is constructed from it as the agent's context. Instantiated on finite element analysis and evaluated on six benchmark problems, MarkovSim passed 16 of 18 runs and corrected its own faults in both the solver operations and the model physics. The engineer was consulted only where engineering judgement was needed. The context remained bounded throughout every run, regardless of its length.
BibTeX
@misc{lei2026markovsim,
  author = {Yeui-Lung Lei and Wen-Bo Chen and Pan Zhang},
  title = {{MarkovSim: an LLM Agent Harness with Bounded Context for Long-Horizon Computational Simulation}},
  year = {2026},
  eprint = {2610.35349},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.35349},
}
Figure