Abstract
In the early stages of designing graphical user interfaces (GUIs), sketching is an effective medium for exploring visual appearance, but it often fails to capture interactive "feel." Traditionally, designers must rely on verbal descriptions or high-fidelity prototypes to convey behavior, both of which have significant drawbacks. We present an extension to Apparition, a crowd-powered prototyping tool, that introduces a "demonstrate-and-remix" workflow. By leveraging a collective of "Wizards-of-Oz," the system allows users to create functional interactive behaviors in real-time. Crowd workers record manual canvas operations and subsequently refine them using a timeline-based remixing interface. This approach bridges the gap between static sketches and functional prototypes, enabling rapid iteration without the need for manual coding.
Introduction
The "fidelity gap" in UI design refers to the disparity between the ease of sketching a visual element and the difficulty of making that element interactive. While a button can be drawn in seconds, defining its logic—such as how it responds to collisions or complex user inputs—usually requires specialized tools or programming. This overhead often stifles the rapid, agile exploration necessary in early-stage design.
This research addresses this gap by scaling the "Wizard-of-Oz" technique. Instead of a single researcher mimicking a system's logic, we utilize a real-time crowd to demonstrate and automate behaviors on the fly, allowing designers to maintain the low-friction benefits of sketching while gaining the benefits of interactive prototypes.
The Demonstrate-and-Remix Workflow
The core of the system is a two-phase process that allows crowd workers to translate verbal user requests into repeatable UI logic:
- Demonstrate: When a user describes a desired behavior (e.g., "The character should jump when clicked"), crowd workers manually perform the necessary actions on a shared digital canvas. The system records these operations—including translations, rotations, creations, and deletions—in real-time.
- Remix: Recorded demonstrations are treated as "visual blocks" within a timeline editor. Similar to a DJ editing audio samples, workers can remix these behaviors by adjusting timing, reversing actions, or looping sequences. This allows a rough human demonstration to be polished into a crisp, repeatable interactive behavior.
To ensure consistency and clarity, the system defines three specific properties for every behavior:
- P1 (Trigger): The event that initiates the behavior (e.g., a collision or click).
- P2 (Visual Change): The actual animation or state change recorded on the canvas.
- P3 (Abstract State): Non-visible logic or relationships, such as gravity or internal variables (e.g., "is the character alive?").
Collective Intelligence and Coordination
Operating a real-time crowd-powered system presents unique coordination challenges. The system implements a collective Wizard-of-Oz model, where multiple workers collaborate concurrently. To prevent conflicts, a "locking mechanism" ensures that workers do not overwrite each other's work while demonstrating or remixing.
Furthermore, the system addresses the issue of "worker dynamicity" (churn). By documenting behaviors through the P1-P3 framework, the system maintains a "collective memory." This documentation serves as a tutorial for new workers joining a session, ensuring that the prototype’s logic remains consistent even as the workforce changes.
Results and Discussion
The system was evaluated through complex, multi-element interaction scenarios, such as a "Super Mario" mockup. In these tests, workers successfully coordinated to manage character movement, enemy interactions, and state-dependent animations (like a "death" sequence upon collision).
Key findings include:
- Efficiency: Functional mock-ups could be created in "tens of seconds," matching the speed of live design conversations.
- Fidelity: The remixing interface allowed workers to produce mid-fidelity results from low-fidelity human input by decoupling the performance of a behavior from its execution.
Limitations and Future Work
While effective, the system's performance is currently tied to the expertise of the crowd workers; those unfamiliar with timeline-based editing may experience a learning curve. Additionally, while the P1-P3 framework helps mitigate worker churn, the temporary nature of crowd labor remains a challenge for long-term projects.
Future research aims to use these crowd-generated demonstrations as training data for machine learning models. The ultimate goal is to automate the transition from verbal descriptions to functional UI elements, eventually reducing or removing the need for human intervention in the prototyping loop.
Summary updated