Introduction
Automatic text summarization is often viewed through a binary lens: it is either a fully manual, cognitively demanding task for humans or a fully automated process handled by AI. However, both extremes have significant drawbacks. Manual summarization is inefficient, while fully automated systems frequently struggle with nuance, hallucinate facts, or fail to capture a specific user's "voice."
Researchers from the University of Washington and Dataminr have identified a critical "middle ground"—interactive summarization. This paper maps the design space of human-AI collaboration, proposing a systematic framework to understand how users want to interact with AI models to produce high-quality summaries.
A Taxonomy of Human-AI Interaction
To categorize the diverse ways humans and AI can co-create, the researchers conducted a systematic literature review of 70 academic papers. This resulted in a taxonomy of five distinct interaction patterns, ranging from high-level guidance to granular assistance.
- Guiding Model Output (A): The human sets parameters such as length, style, or specific focus areas before the AI generates the summary.
- Selecting or Rating Model Output (B): The AI generates multiple candidate summaries, and the human selects or ranks the best options.
- Post-editing (C): The AI provides an initial draft, which the human then manually corrects or refines.
- Interactive Editing (D): A dynamic feedback loop where the AI updates the remainder of a summary in real-time based on specific edits the human makes to the beginning.
- Writing with Model Assistance (E): The human takes the lead in writing, while the AI provides inline suggestions or auto-completions, similar to modern predictive text features.
Methodology: The Wizard-of-Oz Study
To evaluate these interactions without the confounding variables of current Large Language Model (LLM) limitations (like latency or specific model quirks), the researchers employed a "Wizard-of-Oz" approach. In this setup, 16 participants interacted with prototypes where a human researcher simulated the AI's responses.
Participants tested these workflows across three diverse datasets to represent varying stakes and complexities:
- Online News: Standard articles from CNN/Daily Mail.
- Social Media: Informal and opinionated posts from Reddit (r/scam and r/wallstreetbets).
- Legal/Technical: Complex U.S. Government bills from the BillSum dataset.
Results: The Efficiency-Control-Trust Trilemma
The study revealed that user preference is rarely about finding a single "best" method, but rather about balancing three competing factors: efficiency, control, and trust. The researchers visualized these trade-offs using radar charts for each interaction type.
Key findings from this mapping include:
- The "Final Say" Requirement: Regardless of the workflow, users almost universally demanded the ability to post-edit. Trust was rarely high enough for users to let the AI have the final word without human oversight.
- Guiding vs. Doing: "Guiding Model Output" scored high for trust because users felt a clear causal link between their instructions and the result. Conversely, "Selecting/Rating" was efficient but left users feeling mentally detached from the final product.
- The Stress of Automation: "Interactive Editing" proved polarizing. While some found the real-time updates helpful, others found it stressful to have the text shifting dynamically while they were trying to focus.
Qualitative Insights and Agency
The research highlighted that trust in AI is not just about the quality of the output, but the transparency of the logic. Users expressed a desire for the AI to highlight which parts of the source text it prioritized, allowing them to verify the summary's accuracy more quickly.
Furthermore, the level of desired agency was context-dependent. For dense, "dry" legal bills, users were happy to prioritize efficiency and let the AI lead. However, for social media posts where "voice" and perspective are paramount, users demanded higher levels of control to ensure the summary remained faithful to the original author's intent.
Conclusion and Future Directions
This research provides a foundational map for designers building AI-powered writing tools. The findings suggest that future systems should not be one-size-fits-all. Instead, they should be adaptive—offering high-automation workflows for routine tasks and high-control interfaces for high-stakes or creative work. By understanding the trade-offs between efficiency, control, and trust, developers can create collaborative environments where humans and AI play to their respective strengths.
Summary updated