Abstract
While natural language prompts have become the standard interface for Generative AI (GenAI), they often suffer from ambiguity and redundancy, creating a "gulf of execution" for users with complex intents. This paper proposes a systematic framework for "interaction-augmented instructions"—a design approach that supplements text with user interface actions such as brushing, clicking, or flowcharting. By analyzing 52 existing systems, the authors establish the 4W1H framework (Why, When, Who, What, How) to categorize how these hybrid interactions help users more precisely restrict, expand, organize, and refine their communication with AI models.
Introduction
The rise of Large Language Models (LLMs) has popularized natural language as the primary mode of human-computer interaction. However, the inherent messiness of human language often makes it difficult for users to translate their mental models into precise instructions. Traditional User Interfaces (WIMP: Windows, Icons, Menus, Pointers) offer precision but lack flexibility, while text prompts offer flexibility but lack precision.
Recent commercial and academic tools—such as OpenAI’s Canvas or the Cursor code editor—have begun to bridge this gap by allowing users to interact directly with content to guide the AI. Despite these advancements, the design space remains fragmented. This research provides a comprehensive "panorama" of the field, offering a systematic design language to help developers integrate interaction into the prompting process.
The 4W1H Framework
To categorize the diverse landscape of interaction-augmented prompting, the authors developed a two-part theoretical contribution. First, they adapted the classic Instrumental Interaction Model for GenAI, where the AI acts as a translator interpreting a combination of text and UI modalities to operate on domain objects.
Second, they introduced the 4W1H Framework, which evaluates tools across five dimensions:
- Why (Purpose): Does the interaction help the user Restrict the AI's scope, Expand context, Organize complex logic, or Refine previous outputs?
- When (Timing): Does the interaction occur during the initial setup or as a follow-up refinement?
- Who (Initiator): Is the interaction human-initiated (e.g., a user brushing an image) or AI-initiated (e.g., the AI providing a widget for the user to select an option)?
- What (Modality): What specific UI elements are used (e.g., sketches, sliders, bounding boxes)?
- How (Application): Does the interaction operate directly on the object or indirectly by calling external tools?
Analysis and Key Insights
The researchers applied this framework to a corpus of 52 research systems and commercial tools published between 2022 and 2024, including projects from top-tier HCI venues like CHI, UIST, and IUI. Several key insights emerged from this meta-analysis:
The Dominance of "Restricting"
Over half of the surveyed tools (29/52) focus on Restricting instructions. This suggests that "over-generation" and vague scope are the primary pain points in current GenAI workflows. Users frequently use visual tools like bounding boxes or text highlighting to tell the AI exactly which part of a document or image to modify, effectively narrowing the AI's focus to reduce errors.
The Four Pillars of Intent
The framework identifies four primary reasons for augmenting text with interaction:
- Restrict: Limiting the AI’s focus to a specific spatial or semantic area.
- Expand: Providing non-textual context, such as a reference video or a hand-drawn sketch, that is difficult to describe in words.
- Organize: Structuring unstructured prompts into hierarchies or flowcharts to help the AI process complex, multi-step logic.
- Refine: Using iterative feedback loops to clarify ambiguous outputs.
The Shift Toward AI-Initiated Interaction
A significant trend is the rise of AI-initiated post-invocation. In these systems, the AI generates fine-grained options or widgets after the initial prompt is sent. This allows users to "pick" from a set of valid paths rather than struggling to describe their next step from scratch. This shift reduces the cognitive load on the user and helps bridge the gap between intent and execution.
Discussion and Future Directions
While interaction-augmented instructions offer clear benefits, the authors identify several trade-offs and areas for future research:
- The Semantic Trade-off: While clicking is more precise for spatial tasks, natural language remains more efficient for broad semantic tasks (e.g., "select all the red cars"). Determining the optimal "switching point" between these modalities is a key challenge for UI designers.
- Human Agency vs. Automation: There is an inherent tension between automation and control. Indirect operations, where the AI calls other tools autonomously, increase efficiency but can leave the user feeling disconnected from the process.
- Design Gaps: The analysis revealed "empty spaces" in the current research landscape. For instance, there are very few tools where the AI proactively helps a user organize their instructions, representing a significant opportunity for future development.
Conclusion
As GenAI moves beyond simple chat interfaces, the need for more sophisticated, multi-modal prompting becomes clear. The 4W1H framework provides a robust foundation for designing the next generation of AI tools, moving us toward a future where human intent is captured not just through words, but through a seamless blend of text and intuitive interaction.