Abstract
User evaluation is the gold standard for visualization research, but it is often slow, expensive, and difficult to scale. With the rise of Large Language Models (LLMs) and their increasing visual literacy, researchers are exploring whether AI agents can serve as low-cost proxies for human subjects. This paper presents an empirical study investigating the alignment between LLM agents (specifically GPT-4V and GPT-4o) and human subjective ratings across various visualization tasks. While agents show promise in mimicking "average" human reasoning for straightforward tasks, they struggle with nuanced aesthetic judgments and fail to capture the diversity of human perspectives.
Introduction
In visualization research, subjective ratings—such as "ease of use," "confidence," and "aesthetic appeal"—are critical for assessing usability. However, conducting large-scale human studies is a significant bottleneck. While previous research has demonstrated that LLMs can handle objective tasks, such as extracting values from a bar chart, their ability to replicate subjective human experience remains largely untested. This study addresses whether an LLM’s "opinion" on a visualization aligns with human sentiment and explores the conditions under which AI can—and cannot—replace human participants.
Methodology
The researchers conducted a multi-stage empirical study to evaluate and improve agent-human alignment:
- Study I (Characteristics): A replication of a classic time-series visualization study using "chain-of-thought" prompting. Agents were required to explain their internal knowledge of a chart and outline task steps before providing a rating.
- Study II (Alignment): A systematic replication of six experiments from five prominent public papers. To gauge predictability, five visualization experts coded their "confidence" in the original hypotheses to see if expert certainty correlated with LLM alignment.
- Study III (Enhancement): An exploration of engineering strategies to improve alignment, including Retrieval-Augmented Generation (RAG) to provide domain-specific context and the manipulation of visual versus textual cues.
- Fast Prototyping Case Study: A real-world application where agents were used to "pre-test" a new experimental design before a formal human study was conducted.
Experimental Datasets
The study utilized experimental materials and results from five public studies hosted on the Open Science Framework (OSF):
- Timeline Shapes: Evaluating the effectiveness of horizontal vs. spiral timelines.
- Imputation for Uncertainty: Assessing confidence levels in missing data representations.
- Fit Estimation: Judging the correlation and fit of data points to a regression line.
- Texture: Rating the visual appeal and readability of different textures.
- Magnitude Judgment: Analyzing how axis limits influence the perception of data values.
Key Insights and Findings
The "Average User" Trap
One of the most significant findings is that LLM agents tend to act as an aggregation of their training data, effectively simulating an "average" user. When researchers attempted to change the agent's persona (e.g., varying age, gender, or education level) in the prompt, it had almost no effect on the resulting ratings. This suggests that current LLMs are unable to capture the diversity of human perspectives inherent in subjective testing.
Expert Confidence as a Predictor
The study revealed a strong correlation between expert consensus and LLM performance. If visualization experts were highly confident about a specific principle (e.g., that horizontal timelines are more readable), the LLM was likely to align with human data. However, when tasks were nuanced or experts were divided, the LLM’s alignment dropped significantly.
Textual Bias and RAG
LLMs remain "text-first" thinkers. The researchers found that providing detailed text descriptions without accompanying images sometimes led to more consistent ratings, though these were often based on stereotypes rather than visual perception. Conversely, Retrieval-Augmented Generation (RAG) proved highly effective; injecting specific domain knowledge (e.g., explaining the hierarchy of an icicle plot) significantly improved the agent's reasoning in complex scenarios.
Aesthetics: The AI Blind Spot
While agents performed well on "obvious" tasks, they failed consistently in simulating human ratings for aesthetic appeal or textural readability. In these areas, agents often provided generic or even "anti-human" interpretations, highlighting a major limitation in using AI for design-centric evaluations.
Practical Implications: Fast Prototyping
In the fast-prototyping case study, agent ratings correctly predicted the trend of results (e.g., R1 > R2 > R3) for a new experiment. This was later confirmed by a 38-participant human study. This suggests that while LLMs may not be ready to provide absolute values or replace formal studies, they are excellent tools for trend prediction and pre-testing experimental designs.
Limitations and Future Work
The study identifies several hurdles for the future of AI-driven visualization research:
- Data Pollution: The risk that LLMs were exposed to the original research papers during training remains a concern.
- Static Scope: The current study was limited to 2D static visualizations; interactive and 3D systems remain unexplored.
- Diversity: Future work should focus on developing "profile-aware" agents, perhaps by fine-tuning models on actual user study logs to better simulate human diversity.
Conclusion
LLM agents like GPT-4V are promising "co-pilots" for visualization designers, offering a way to quickly iterate on hypotheses and predict general trends. However, they are not yet a substitute for human subjects, particularly when it comes to nuanced, aesthetic, or diverse subjective experiences.
Summary updated