TL;DR
This paper explores a paradigm shift in human-AI collaboration from rigid, turn-based delegation to fluid, concurrent co-creation within a shared digital workspace. By developing Cleo, a context-aware AI design agent for Figma, the researchers demonstrate that making an agent's execution process transparent prompts users to work alongside it simultaneously. Their findings show that concurrent interaction occurred in nearly a third of all work sessions, positioning real-time collaboration not as a replacement for delegation, but as a way to make delegation work better.
What Problem Does This Paper Address?
Most current generative AI tools and agents operate on a strict "wait-your-turn" prompt-and-response cycle. While this works for simple automation, it creates significant friction in creative and complex workflows like UI/UX design.
As AI agents become more autonomous and execute multi-step tasks directly inside shared workspaces (like Figma), their intermediate actions become visible to users. The authors discovered that this process transparency naturally makes users want to jump in and edit the canvas mid-execution. However, existing agents lack collaborative context awareness—they cannot distinguish whether a user's real-time edit is feedback meant to redirect the agent, or independent parallel work that should be left alone. This lack of awareness leads to collaborative friction, where agents accidentally overwrite or misinterpret human contributions.
Methods
The researchers tackled this coordination problem through a two-stage technology probe study:
- Study 1 (Initial Probe): They built a process-transparent Figma plugin using a ReAct (Reasoning and Acting) architecture powered by Claude 4.5 Sonnet and Claude 4.5 Haiku. This agent exposed its step-by-step design process directly on the canvas. Testing with 10 designers confirmed that visibility encouraged real-time intervention but highlighted the agent's inability to handle simultaneous human actions.
- System Redesign (Cleo): To give the agent collaborative context awareness, they developed Cleo (Collaborative Linked Executive Operator). Cleo introduced three novel technical modules:
- User Change Detection Module: Analyzes canvas modifications made by the user while the agent is idle.
- Attribution Change Module: Distinguishes between agent actions and user actions occurring simultaneously during execution.
- Plan Update Module: Dynamically updates the agent's high-level plan only when a clear conflict with the user's intent is detected.
- Study 2 (Evaluation): They conducted a two-day exploratory study with 10 professional UI/UX designers using Cleo on both closed- and open-ended design tasks. They utilized stimulated recall interviews, where participants reviewed visualized logs of their interaction dynamics to explain why they intervened at specific moments.
Datasets
The paper introduces an annotated dataset of 214 turn-level human-agent co-creative interactions collected during Study 2. The dataset uniquely captures concurrent interaction logs, detailing exactly when and how users chose to work alongside the agent versus when they opted for hands-off delegation.
Key Insights and Ideas
- Process Visibility Shifts User Expectations: When users can see an AI's intermediate steps, they stop treating it as an opaque black box. Instead of waiting passively for a final output, they actively use the agent's visible workflow to plan their own next steps or catch mistakes early.
- The Unique Affordance of Human-Agent Asymmetry: Unlike human-to-human collaboration, collaborating with an AI agent allows users to interrupt without guilt, abruptly terminate tasks when they get tedious, or appropriate partial outputs without attribution. Designers freely exploited this asymmetry to fuel highly experimental, rapid workflows.
- Observation is an Active Strategy: Passive monitoring (the "Observational" mode) isn't just dead time. Users strategically watch the agent to time their own parallel tasks, stepping in only when an "Idea Spark" occurs or when the agent's quality drops.
Key Results
The authors developed a comprehensive taxonomy of human-agent interaction, identifying 5 main action categories, 6 triggers for intervention, and 4 enabling factors. These were synthesized into a grounded Decision Model mapping how users navigate collaboration loops.
- High Prevalence of Synchronous Work: Concurrent interaction (where humans and AI worked on the canvas simultaneously) appeared in 31.8% of all interaction turns.
- Full Delegation is Dynamic: Immediate hands-off delegation occurred in 31.3% of turns, but users frequently cycled between full delegation, active observation, and direct intervention based on changing task priorities.
- Taxonomy of Concurrent Actions: The authors identified distinct behavioral patterns within concurrent work, such as In-situ co-editing (working on the same subtask), Opportunistic takeover (finishing a pending subtask while the agent handles another), and Demonstration-based steering (showing the AI what to do via direct manipulation rather than text prompts).
Limitations and Future Work
While promising, the study was limited to 10 participants over a two-day period, meaning larger-scale, longitudinal studies are required to see how these interaction patterns evolve as users develop long-term mental models of the AI. Additionally, the current system is highly optimized for visual, canvas-based environments like Figma. The authors note that extending concurrent interaction to text-heavy domains (like coding or writing) will require new multi-scale representations (e.g., hierarchical text summaries and lightweight annotations) so humans can keep pace with the rapid speed of AI generation.
Summary updated