Introduction
As Large Language Models (LLMs) become central to workflows in research, development, and creative industries, users increasingly find themselves generating dozens or even hundreds of responses to the same prompt. Whether auditing a model for bias, comparing the performance of two different architectures, or brainstorming creative concepts, users face a significant cognitive challenge: the "mesoscale."
Researchers from Harvard and the University of Sydney define the mesoscale as the range of 10 to 100+ outputs—a volume too large for sequential reading in a standard chat interface, yet too small to require fully automated statistical summarization. This paper introduces a novel exploratory interface designed to help users navigate this space by visualizing patterns, similarities, and outliers across large sets of LLM data.
The Problem: Beyond the Linear Chat
Traditional LLM interfaces, such as ChatGPT, are built for linear, one-by-one interactions. When a user needs to evaluate 50 variations of a response, they often resort to "hacking" the process by copying text into massive spreadsheets. This approach is arduous and clunky, leading to a "wall of text" effect that obscures the nuances of model behavior. The authors argue that users need better "skimming support" that leverages spatial memory and highlights the distribution of possible answers rather than just the single best result.
Methodology and System Design
The team developed an interface featuring five key visualization and analysis techniques to reduce cognitive load:
- Grid Layout: Instead of a vertical list, responses are organized in a spatial grid. This allows users to arrange outputs by variables (e.g., Model A vs. Model B) and use spatial memory to track specific responses.
- Exact Matches: The system automatically highlights identical substrings across different responses, allowing users to instantly see where the model is being consistent or repetitive.
- Unique Words: Using TF-IDF (Term Frequency-Inverse Document Frequency), the tool highlights words that are distinctive to a specific response, helping outliers stand out.
- Positional Diction Clustering (PDC): The paper’s primary technical contribution is PDC, an agglomerative clustering algorithm. It groups sentences from different responses that are both semantically similar and located in roughly the same relative position within the text.
- Interleaved View: This view renders the results of PDC by stacking similar sentences together. Redundant text is grayed out, allowing users to "read across" the variations of a specific point or argument without re-reading the entire document.
Evaluation
The researchers evaluated the tool through a controlled study with 24 participants and eight in-depth case studies. The controlled study involved two tasks: an email rewriting task and a model comparison task (GPT-3.5 vs. GPT-4) using advice-seeking prompts. The case studies involved professionals in law, journalism, and creative writing applying the tool to real-world problems, such as identifying trademark confusion and auditing historical agency in model outputs.
Key Insights and Results
The evaluation yielded several significant findings regarding how users interact with LLM outputs at scale:
- Enhanced Discovery: In model comparison tasks, participants using the exploratory interface identified significantly more differences between models ($p < 0.01$) than those using a standard linear list. The tool made it easier to spot subtle shifts in "voice" or systematic hallucinations.
- The "Shopping Cart" Metaphor: One of the most compelling qualitative findings was the emergence of a "shopping cart" workflow. Users in the case studies used the Interleaved PDC view to pick and choose the best-phrased sentences from various outputs, effectively "remixing" multiple responses into a single, optimized final version.
- Engagement vs. Speed: While the exploratory interface took slightly longer to use on average (9.7 minutes vs. 8.8 minutes), this was attributed to higher user engagement. Participants spent more time exploring nuances they would have otherwise missed in a list view.
- Subjective Preference: Participants overwhelmingly preferred the exploratory interface ($p < .05$), specifically citing the PDC algorithm as the most useful feature for skipping redundant information.
Limitations and Future Work
While the system excels at the mesoscale, the authors note that scaling to the "macro" level (1,000+ responses) would require further abstraction, such as automated sentiment buckets or toxicity scores. Additionally, the PDC algorithm can struggle if there is extreme variation in the text, and some users reported a loss of "narrative flow" when reading sentences in the interleaved view. Future research will focus on integrating more automated analytical "lenses" and improving context retention during clustering.
Conclusion
By moving beyond the spreadsheet and the chat box, this exploratory interface provides a necessary framework for understanding LLM behavior at scale. The introduction of Positional Diction Clustering offers a powerful way to synthesize multiple outputs, turning a "wall of text" into a structured, navigable landscape for researchers and creators alike.
Summary updated