Creating Interactive Behaviors in Early Sketch by Recording and Remixing Crowd Demonstrations

Sang Won Lee, Yiwei Yang, Shiyan Yan, Yujin Zhang, Isabelle Wong, Zhengxi Tan, Miles McGruder, Christopher Homan, Walter S. LaseckiarXiv 2016 · OpenReview

Abstract

In the early stages of designing graphical user interfaces (GUIs), sketching is an effective medium for exploring visual appearance, but it often fails to capture interactive "feel." Traditionally, designers must rely on verbal descriptions or high-fidelity prototypes to convey behavior, both of which have significant drawbacks. We present an extension to Apparition, a crowd-powered prototyping tool, that introduces a "demonstrate-and-remix" workflow. By leveraging a collective of "Wizards-of-Oz," the system allows users to create functional interactive behaviors in real-time. Crowd workers record manual canvas operations and subsequently refine them using a timeline-based remixing interface. This approach bridges the gap between static sketches and functional prototypes, enabling rapid iteration without the need for manual coding.

Introduction

The "fidelity gap" in UI design refers to the disparity between the ease of sketching a visual element and the difficulty of making that element interactive. While a button can be drawn in seconds, defining its logic—such as how it responds to collisions or complex user inputs—usually requires specialized tools or programming. This overhead often stifles the rapid, agile exploration necessary in early-stage design.

This research addresses this gap by scaling the "Wizard-of-Oz" technique. Instead of a single researcher mimicking a system's logic, we utilize a real-time crowd to demonstrate and automate behaviors on the fly, allowing designers to maintain the low-friction benefits of sketching while gaining the benefits of interactive prototypes.

The Demonstrate-and-Remix Workflow

The core of the system is a two-phase process that allows crowd workers to translate verbal user requests into repeatable UI logic:

  1. Demonstrate: When a user describes a desired behavior (e.g., "The character should jump when clicked"), crowd workers manually perform the necessary actions on a shared digital canvas. The system records these operations—including translations, rotations, creations, and deletions—in real-time.
  2. Remix: Recorded demonstrations are treated as "visual blocks" within a timeline editor. Similar to a DJ editing audio samples, workers can remix these behaviors by adjusting timing, reversing actions, or looping sequences. This allows a rough human demonstration to be polished into a crisp, repeatable interactive behavior.

To ensure consistency and clarity, the system defines three specific properties for every behavior:

  • P1 (Trigger): The event that initiates the behavior (e.g., a collision or click).
  • P2 (Visual Change): The actual animation or state change recorded on the canvas.
  • P3 (Abstract State): Non-visible logic or relationships, such as gravity or internal variables (e.g., "is the character alive?").

Collective Intelligence and Coordination

Operating a real-time crowd-powered system presents unique coordination challenges. The system implements a collective Wizard-of-Oz model, where multiple workers collaborate concurrently. To prevent conflicts, a "locking mechanism" ensures that workers do not overwrite each other's work while demonstrating or remixing.

Furthermore, the system addresses the issue of "worker dynamicity" (churn). By documenting behaviors through the P1-P3 framework, the system maintains a "collective memory." This documentation serves as a tutorial for new workers joining a session, ensuring that the prototype’s logic remains consistent even as the workforce changes.

Results and Discussion

The system was evaluated through complex, multi-element interaction scenarios, such as a "Super Mario" mockup. In these tests, workers successfully coordinated to manage character movement, enemy interactions, and state-dependent animations (like a "death" sequence upon collision).

Key findings include:

  • Efficiency: Functional mock-ups could be created in "tens of seconds," matching the speed of live design conversations.
  • Fidelity: The remixing interface allowed workers to produce mid-fidelity results from low-fidelity human input by decoupling the performance of a behavior from its execution.

Limitations and Future Work

While effective, the system's performance is currently tied to the expertise of the crowd workers; those unfamiliar with timeline-based editing may experience a learning curve. Additionally, while the P1-P3 framework helps mitigate worker churn, the temporary nature of crowd labor remains a challenge for long-term projects.

Future research aims to use these crowd-generated demonstrations as training data for machine learning models. The ultimate goal is to automate the transition from verbal descriptions to functional UI elements, eventually reducing or removing the need for human intervention in the prototyping loop.

Summary updated

Abstract

In the early stages of designing graphical user interfaces (GUIs), the look (appearance) can be easily presented by sketching, but the feel (interactive behaviors) cannot, and often requires an accompanying description of how it works (Myers et al. 2008). We propose to use crowdsourcing to augment early sketches with interactive behaviors generated, used, and reused by collective "wizards-of-oz" as opposed to a single wizard as in prior work (Davis et al. 2007). This demo presents an extension of Apparition (Lasecki et al. 2015), a crowd-powered prototyping tool that allows end users to create functional GUIs using speech and sketch. In Apparition, crowd workers collaborate in real-time on a shared canvas to refine the user-requested sketch interactively, and with the assistance of the end users. Our demo extends this functionality to let crowd workers "demonstrate" the canvas changes that are needed for a behavior and refine their demonstrations to improve the fidelity of interactive behaviors. The system then lets workers "remix" these behaviors to make creating future behaviors more efficient.
BibTeX
@misc{lee2016creating,
  author = {Sang Won Lee and Yiwei Yang and Shiyan Yan and Yujin Zhang and Isabelle Wong and Zhengxi Tan and Miles McGruder and Christopher Homan and Walter S. Lasecki},
  title = {{Creating Interactive Behaviors in Early Sketch by Recording and Remixing Crowd Demonstrations}},
  year = {2016},
  eprint = {1609.01382},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1609.01382},
}
Figure