TL;DR
SWE-WhatIf is a framework that turns recorded developer-agent sessions into dynamic evaluation tasks. By modeling a user’s underlying intents and satisfaction criteria rather than just replaying static messages, it creates a simulator that can "faithfully adapt" to new agent trajectories while remaining grounded in the original user's goals and style.
What Problem Does This Paper Address?
Evaluating coding agents (like GitHub Copilot Workspace or OpenDevin) is notoriously difficult. Static benchmarks often lack the interactive "steering" that characterizes real-world development. While researchers have begun releasing logs of real human-agent sessions, these recordings are "brittle."
If you try to test a new agent by replaying a recorded session, the new agent will inevitably take a different path—perhaps it fixes a bug faster or makes a different mistake. In these cases, the recorded user's next message often becomes nonsensical or irrelevant. To truly evaluate how an agent handles real-world work, we need a user simulator that can react to an agent's novel actions while still behaving like the specific human from the original recording.
Methods
The core innovation of SWE-WhatIf is the Intent Model. Instead of treating a session as a linear script, the framework uses an LLM-based pipeline to extract:
- Intents: The specific outcomes or information the user wanted (e.g., "Fix the login bug").
- Criteria: The specific rules for satisfying an intent (e.g., "The fix must not break existing tests").
- Conditions: The triggers that cause an intent to be raised or dropped (e.g., "Only ask for a regression test if the initial fix fails").
During a replay, the simulator follows a three-step loop for every agent turn:
- Assess: An LLM judge evaluates the agent’s latest action against the active intent criteria.
- Update: The simulator updates the state of each intent (e.g., from blocked to open, or raised to satisfied).
- Respond: An LLM generates a reply based on the current intent state, filtered through a Style Profile (capturing the original user's formality, length, and punctuation habits).
Datasets
The authors evaluated SWE-WhatIf using 89 real-world sessions curated from existing datasets:
- SWE-chat: Interactions between developers and coding agents in the wild.
- DataClaw: A dataset of agent chat histories.
- SWE-Together: A collection of sessions with pre-built starting environments.
The tasks spanned multiple programming languages (primarily English, with some Chinese and Vietnamese) and required the agent to operate within a reconstructed repository environment.
Key Insights and Ideas
- Faithful Adaptability: The authors define this as the ability to respond to actions absent from the recording, but only in ways supported by the user’s original requests. This prevents the simulator from becoming a "generic" user and keeps the evaluation specific to the original task.
- Steering Matters for Ranking: The study found that agent rankings change depending on how a user steers. For example, GPT-6 Sol performed best when requirements were stated upfront (detailed opening requests), while Claude models handled "reactive" intents (requirements raised later in response to agent errors) more effectively.
- The "Burden-Adjusted Success" Metric: To prevent agents from succeeding simply by badgering the user for infinite hints, the authors introduced a metric that penalizes success if it requires more turns than the original human session.
Key Results
- Replication & Robustness: SWE-WhatIf generated replies more similar to real users than baselines like SWE-Together or SWE-Interact, even when the original sessions were paraphrased to test for wording bias.
- Adaptation Win Rate: In tests where agents were intentionally instructed to avoid mistakes the original user pushed back on, SWE-WhatIf correctly rewarded the "proactive" agent with a 66.3% win rate, significantly outperforming other simulators.
- Human Utility: In a study with 16 developers replaying their own sessions, 14 out of 16 participants changed their ranking of which agent was best after seeing the simulated replays. Many realized that smaller, cheaper models were often sufficient for their specific tasks.
Limitations and Future Work
- Chat-Only Interaction: The simulator assumes the user only interacts via text. In reality, developers often edit files or run tests themselves mid-session, which isn't currently captured.
- Implicit Acceptance Bias: The model assumes that if a user didn't object to an outcome in the recording, they were satisfied with it. This ignores cases where a user might have been frustrated but stayed silent.
- Evidence Constraints: The simulator only asks for things the user explicitly requested in the logs. It cannot currently simulate a user coming up with a new request that might have arisen if the agent had taken a completely different (but valid) technical path.
Summary updated