SWE-WhatIf: Simulating Real Users of Coding Agents from Recorded Session

Tae Soo Kim, Tongshuang Wu, Juho KimarXiv 2026 · arXiv

TL;DR

SWE-WhatIf is a framework that turns recorded developer-agent sessions into dynamic evaluation tasks. By modeling a user’s underlying intents and satisfaction criteria rather than just replaying static messages, it creates a simulator that can "faithfully adapt" to new agent trajectories while remaining grounded in the original user's goals and style.

What Problem Does This Paper Address?

Evaluating coding agents (like GitHub Copilot Workspace or OpenDevin) is notoriously difficult. Static benchmarks often lack the interactive "steering" that characterizes real-world development. While researchers have begun releasing logs of real human-agent sessions, these recordings are "brittle."

If you try to test a new agent by replaying a recorded session, the new agent will inevitably take a different path—perhaps it fixes a bug faster or makes a different mistake. In these cases, the recorded user's next message often becomes nonsensical or irrelevant. To truly evaluate how an agent handles real-world work, we need a user simulator that can react to an agent's novel actions while still behaving like the specific human from the original recording.

Methods

The core innovation of SWE-WhatIf is the Intent Model. Instead of treating a session as a linear script, the framework uses an LLM-based pipeline to extract:

  1. Intents: The specific outcomes or information the user wanted (e.g., "Fix the login bug").
  2. Criteria: The specific rules for satisfying an intent (e.g., "The fix must not break existing tests").
  3. Conditions: The triggers that cause an intent to be raised or dropped (e.g., "Only ask for a regression test if the initial fix fails").
Figure 1: Example simulated replays of a recorded ses- sion with three agents. SWE-WHATIF builds an intent model of the user from the recorded session (top), and the simulated user replies to each agent’s turn from this model (bottom).

During a replay, the simulator follows a three-step loop for every agent turn:

  • Assess: An LLM judge evaluates the agent’s latest action against the active intent criteria.
  • Update: The simulator updates the state of each intent (e.g., from blocked to open, or raised to satisfied).
  • Respond: An LLM generates a reply based on the current intent state, filtered through a Style Profile (capturing the original user's formality, length, and punctuation habits).
Figure 2: After Agent B’s turn in Figure 1, the simulator responds by (1) assessing the criteria of the raised intents and the conditions of blocked intents against the agent’s turn, (2) updating the intent states accordingly, and (3) responding by deciding which intents to express in the next message and writing this message in the user’s style.

Datasets

The authors evaluated SWE-WhatIf using 89 real-world sessions curated from existing datasets:

  • SWE-chat: Interactions between developers and coding agents in the wild.
  • DataClaw: A dataset of agent chat histories.
  • SWE-Together: A collection of sessions with pre-built starting environments.

The tasks spanned multiple programming languages (primarily English, with some Chinese and Vietnamese) and required the agent to operate within a reconstructed repository environment.

Key Insights and Ideas

  • Faithful Adaptability: The authors define this as the ability to respond to actions absent from the recording, but only in ways supported by the user’s original requests. This prevents the simulator from becoming a "generic" user and keeps the evaluation specific to the original task.
  • Steering Matters for Ranking: The study found that agent rankings change depending on how a user steers. For example, GPT-6 Sol performed best when requirements were stated upfront (detailed opening requests), while Claude models handled "reactive" intents (requirements raised later in response to agent errors) more effectively.
  • The "Burden-Adjusted Success" Metric: To prevent agents from succeeding simply by badgering the user for infinite hints, the authors introduced a metric that penalizes success if it requires more turns than the original human session.

Key Results

  • Replication & Robustness: SWE-WhatIf generated replies more similar to real users than baselines like SWE-Together or SWE-Interact, even when the original sessions were paraphrased to test for wording bias.
  • Adaptation Win Rate: In tests where agents were intentionally instructed to avoid mistakes the original user pushed back on, SWE-WhatIf correctly rewarded the "proactive" agent with a 66.3% win rate, significantly outperforming other simulators.
Figure 3: Replication and robustness score averages (i.e., similarity) against adaptation score (i.e., win rate of the proactive policy), per user simulator. Filled points are simulators with significant results for adaptation.
  • Human Utility: In a study with 16 developers replaying their own sessions, 14 out of 16 participants changed their ranking of which agent was best after seeing the simulated replays. Many realized that smaller, cheaper models were often sufficient for their specific tasks.
Figure 5: (a) Mean rank that participants gave each agent before reading the replays (start of each arrow) and after (arrowhead), where 1 is best. The number of participants who saw each agent is in parentheses. (b) Participants’ agreement with the four statements about their user simulators and the replays (16 participants).

Limitations and Future Work

  • Chat-Only Interaction: The simulator assumes the user only interacts via text. In reality, developers often edit files or run tests themselves mid-session, which isn't currently captured.
  • Implicit Acceptance Bias: The model assumes that if a user didn't object to an outcome in the recording, they were satisfied with it. This ignores cases where a user might have been frustrated but stayed silent.
  • Evidence Constraints: The simulator only asks for things the user explicitly requested in the logs. It cannot currently simulate a user coming up with a new request that might have arisen if the agent had taken a completely different (but valid) technical path.

Summary updated

Abstract

Recorded sessions between users and coding agents offer a way to evaluate how agents complete real-world tasks and interact with users. However, these sessions cannot simply be replayed with new agents, because recorded user replies may no longer apply when an agent takes a different trajectory. We introduce SWE-WhatIf, which replays recorded sessions with new agents by modeling the user's intents, the criteria for satisfying them, and the conditions under which they arise. Using this model, a user simulator generates replies that adapt to unseen agent turns while remaining grounded in the recording. Across 89 real-world sessions, SWE-WhatIf generated the replies most similar to those of real users and matched the strongest baseline when sessions were rewritten. To test adaptation, we replay sessions with agents instructed to avoid or repeat actions that the recorded user pushed back on. SWE-WhatIf produced the highest win rate for avoiding these actions (66.3%), significantly above chance. We further demonstrate its utility by ranking five LLMs as coding agents, where the highest-ranked model depended on how users specified and developed their intents, and a study with 16 users, where replaying their own sessions changed 14 participants' rankings and informed their future agent use.
BibTeX
@misc{kim2026swe,
  author = {Tae Soo Kim and Tongshuang Wu and Juho Kim},
  title = {{SWE-WhatIf: Simulating Real Users of Coding Agents from Recorded Session}},
  year = {2026},
  eprint = {2611.21095},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.21095},
}
Figure