Mapping the Design Space of Human-AI Interaction in Text Summarization

Ruijia Cheng, Alison Smith-Renner, Ke Zhang, Joel R. Tetreault, Alejandro Jaimes-LarrarteNAACL 2022 · DOI

Introduction

Automatic text summarization is often viewed through a binary lens: it is either a fully manual, cognitively demanding task for humans or a fully automated process handled by AI. However, both extremes have significant drawbacks. Manual summarization is inefficient, while fully automated systems frequently struggle with nuance, hallucinate facts, or fail to capture a specific user's "voice."

Researchers from the University of Washington and Dataminr have identified a critical "middle ground"—interactive summarization. This paper maps the design space of human-AI collaboration, proposing a systematic framework to understand how users want to interact with AI models to produce high-quality summaries.

A Taxonomy of Human-AI Interaction

To categorize the diverse ways humans and AI can co-create, the researchers conducted a systematic literature review of 70 academic papers. This resulted in a taxonomy of five distinct interaction patterns, ranging from high-level guidance to granular assistance.

Figure 1: The five human-AI interaction workflows identified in the study, ranging from guiding model output to writing with model assistance.
  1. Guiding Model Output (A): The human sets parameters such as length, style, or specific focus areas before the AI generates the summary.
  2. Selecting or Rating Model Output (B): The AI generates multiple candidate summaries, and the human selects or ranks the best options.
  3. Post-editing (C): The AI provides an initial draft, which the human then manually corrects or refines.
  4. Interactive Editing (D): A dynamic feedback loop where the AI updates the remainder of a summary in real-time based on specific edits the human makes to the beginning.
  5. Writing with Model Assistance (E): The human takes the lead in writing, while the AI provides inline suggestions or auto-completions, similar to modern predictive text features.

Methodology: The Wizard-of-Oz Study

To evaluate these interactions without the confounding variables of current Large Language Model (LLM) limitations (like latency or specific model quirks), the researchers employed a "Wizard-of-Oz" approach. In this setup, 16 participants interacted with prototypes where a human researcher simulated the AI's responses.

Participants tested these workflows across three diverse datasets to represent varying stakes and complexities:

  • Online News: Standard articles from CNN/Daily Mail.
  • Social Media: Informal and opinionated posts from Reddit (r/scam and r/wallstreetbets).
  • Legal/Technical: Complex U.S. Government bills from the BillSum dataset.

Results: The Efficiency-Control-Trust Trilemma

The study revealed that user preference is rarely about finding a single "best" method, but rather about balancing three competing factors: efficiency, control, and trust. The researchers visualized these trade-offs using radar charts for each interaction type.

Figure 4: Radar charts comparing the five interaction types across the dimensions of Efficiency, Control, and Trust.

Key findings from this mapping include:

  • The "Final Say" Requirement: Regardless of the workflow, users almost universally demanded the ability to post-edit. Trust was rarely high enough for users to let the AI have the final word without human oversight.
  • Guiding vs. Doing: "Guiding Model Output" scored high for trust because users felt a clear causal link between their instructions and the result. Conversely, "Selecting/Rating" was efficient but left users feeling mentally detached from the final product.
  • The Stress of Automation: "Interactive Editing" proved polarizing. While some found the real-time updates helpful, others found it stressful to have the text shifting dynamically while they were trying to focus.

Qualitative Insights and Agency

The research highlighted that trust in AI is not just about the quality of the output, but the transparency of the logic. Users expressed a desire for the AI to highlight which parts of the source text it prioritized, allowing them to verify the summary's accuracy more quickly.

Furthermore, the level of desired agency was context-dependent. For dense, "dry" legal bills, users were happy to prioritize efficiency and let the AI lead. However, for social media posts where "voice" and perspective are paramount, users demanded higher levels of control to ensure the summary remained faithful to the original author's intent.

Conclusion and Future Directions

This research provides a foundational map for designers building AI-powered writing tools. The findings suggest that future systems should not be one-size-fits-all. Instead, they should be adaptive—offering high-automation workflows for routine tasks and high-control interfaces for high-stakes or creative work. By understanding the trade-offs between efficiency, control, and trust, developers can create collaborative environments where humans and AI play to their respective strengths.

Summary updated

BibTeX
@inproceedings{cheng2022mapping,
  author = {Ruijia Cheng and Alison Smith-Renner and Ke Zhang and Joel R. Tetreault and Alejandro Jaimes-Larrarte},
  title = {{Mapping the Design Space of Human-AI Interaction in Text Summarization}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2022},
  doi = {10.18653/v1/2022.naacl-main.33},
}
Figure