TL;DR
ComputerSD is an online training framework that helps computer-use agents (CUAs) learn more effectively by providing fine-grained, real-time feedback instead of relying solely on sparse "success or failure" rewards. By combining a fine-tuned GUI analyzer with a novel "value-gated" self-distillation objective, the method significantly improves agent success rates and out-of-distribution generalization while maintaining high training efficiency through an asynchronous pipeline.
What Problem Does This Paper Address?
Building agents that can navigate complex graphical user interfaces (GUIs) is a major challenge in AI. Most current methods train these agents using Reinforcement Learning (RL) based on outcome rewards—essentially telling the agent "you succeeded" or "you failed" at the end of a long sequence of clicks and keystrokes. This makes "credit assignment" incredibly difficult: if a task fails, was it the first click that was wrong, or the tenth?
While On-Policy Self-Distillation (OPSD) attempts to provide token-level guidance by comparing a "student" agent to a "teacher" with privileged information, it faces two major hurdles in computer-use tasks:
- Misalignment: If guidance is fixed beforehand (like a reference trajectory), it becomes useless the moment the agent takes a different—but potentially still valid—path.
- Unreliable Signals: Even when guidance is relevant, the probability shifts it induces at the token level often contradict the actual correctness of the step. As shown in Figure 1, without regulation, over 80% of tokens at correct steps might be suppressed, and 20% at incorrect steps might be reinforced.
Methods
The authors introduce ComputerSD, which tackles these issues through three main components:
- Real-Time GUI Analyzer: Instead of fixed guidance, they use a lightweight model fine-tuned on expert annotations to analyze GUI transitions as they happen. After every action, the analyzer looks at the "before" and "after" screenshots to provide a step-level value score (was this action good?) and guidance (what should be done next?).
- Value-Gated OPSD: This is the core algorithmic innovation. The agent (student) is compared to a version of itself that has access to the analyzer's guidance (teacher). Crucially, the resulting learning signal is "gated" by the analyzer's value score. If the analyzer judges a step as incorrect, the value gate suppresses the teacher's influence, preventing the model from learning from potentially misleading probability shifts.
- Joint Optimization & Asynchronous Pipeline: ComputerSD optimizes both the token-level OPSD loss and the trajectory-level GRPO (Group Relative Policy Optimization) loss. To prevent the per-step analysis from slowing down training, they developed a fully asynchronous pipeline that overlaps environment interaction, GUI analysis, and policy updates.
Datasets
- OSWorld-Verified: The primary benchmark for training and evaluation, covering diverse real-world computer tasks.
- WindowsAgentArena (WAA): Used to test cross-platform generalization (training on Linux-based OSWorld and testing on Windows).
- GUI Analyzer SFT Dataset: A newly created dataset of 34,188 step-level samples annotated by a strong expert model (Kimi K3) to train the GUI analyzer.
Key Insights and Ideas
- Feedback must be dynamic: The "aha moment" here is that in interactive environments, the environment's response to an action is the best source of truth. Real-time feedback eliminates the misalignment issues inherent in pre-planned guidance.
- Trust, but verify: Even "privileged" guidance can be noisy. The Value Gate acts as a necessary filter, ensuring the agent only internalizes token-level shifts that align with the overall goal of the step.
- Asynchrony is a requirement, not a luxury: Analyzing every single step in a rollout is computationally expensive. The authors show that an asynchronous framework (Figure 3) is vital, increasing training throughput fivefold compared to a synchronous approach.
Key Results
ComputerSD demonstrates clear superiority over standard outcome-only RL:
- Higher Success Rates: On OSWorld-Verified, it improved the success rate of the specialized
EvoCUA-8Bmodel from 43.8% to 47.9%. - Faster Convergence: As seen in Figure 4, ComputerSD reaches the final performance level of standard GRPO using only ~22% to 56% of the trajectory budget, depending on the backbone model.
- Superior Generalization: The gains were most pronounced in "out-of-distribution" tasks (categories not seen during training), suggesting the agent is learning fundamental strategies rather than just memorizing paths.
Limitations and Future Work
The primary limitation is the dependency on the GUI analyzer. If the analyzer makes systematic errors in its value judgments or guidance, it can bias the policy updates. While fine-tuning the analyzer helps, it doesn't eliminate the risk entirely.
Future work could explore ways to improve analyzer reliability without significantly increasing inference costs, perhaps through multi-modal verification or by incorporating uncertainty estimates into the value gate. Additionally, scaling this approach to even larger models and more diverse operating systems remains an open frontier.
Summary updated