Human-Data Interaction, Exploration, and Visualization in the AI Era: Challenges and Opportunities

Jean‐Daniel Fekete, Yifan Hu, Dominik Moritz, Arnab Nandi, Senjuti Basu Roy, Eugene W. Wu, Nikos Bikakis, George Papastefanatos, Panos K. Chrysanthis, Guoliang Li, Lingyun Yu

TL;DR

This paper provides a comprehensive roadmap for the next generation of data interaction systems, arguing that the "AI Era" requires a fundamental shift from static, query-based tools to proactive, guidance-driven ecosystems. The authors call for a "co-design" approach where databases, AI models, and visualization interfaces are built as a single, tightly coupled stack to handle the scale, uncertainty, and multimodal nature of modern data.

What Problem Does This Paper Address?

Traditional data analysis tools were built for a "query-then-explore" workflow: a user knows what they are looking for, writes a structured query (like SQL), and views a static chart. However, we have entered an era where data is increasingly massive, unstructured (video, audio, text), and multimodal.

The authors identify several critical gaps:

  1. Perceptual Latency: Human thought happens in milliseconds, but data systems often take seconds or minutes, disrupting the "flow" of analysis and introducing cognitive bias.
  2. The Multimodal Barrier: You can't query what you don't understand. Skimming a 10,000-hour video archive to find a specific event is humanly impossible without AI "peeking inside" first.
  3. AI Uncertainty: While LLMs and VLMs (Vision-Language Models) are powerful, they are brittle and prone to "hallucinations," making it difficult for users to trust AI-generated insights without better provenance and verification tools.

Methods

As a position paper authored by a cross-disciplinary group of experts, it synthesizes research from databases, HCI, and AI. Rather than a single experiment, it proposes several novel architectural shifts:

  • Interface-System Co-Design: Moving away from treating the UI as a simple "SQL generator." Instead, they propose "interface-aware" system optimizers that understand how a user interacts with a screen to prioritize which data to fetch.
  • Guidance-Driven Exploration: Using foundation models to provide "zero-shot" semantic descriptions and autocompletion for unstructured data (e.g., suggesting sub-queries like "damaged pallets" while a user scans warehouse footage).
  • Generative Visualization: Leveraging Deep Learning (GNNs and GANs) to move beyond hard-coded heuristics for layouts. Instead, systems should "learn" human aesthetic preferences to create more readable and intuitive visualizations.

Datasets

While the paper is a high-level survey and vision document, it references several key benchmarks and data challenges:

  • IDEBench: A benchmark for evaluating interactive data exploration.
  • Multimodal Corpora: The paper discusses the challenges of scaling to massive video archives and visually rich documents.
  • Human Preference Data: The authors highlight a major gap in the field—the lack of large-scale, high-quality datasets of human aesthetic judgments needed to train "aesthetics-aware" AI models.

Key Insights and Ideas

  • The "Streetlight Effect": Current interfaces are often designed based on what existing databases can easily support, rather than what is actually effective for the user. We need systems that can synthesize thousands of potential interface designs based on the analysis task.
  • Visualization as an Active Component: Visualization is shifting from a passive "output" to an active part of the system that mediates trust. It should use narrative and animation to guide a user's attention toward the most reliable AI-generated insights.
  • Cold-Start Visualization: Most "scalable" systems assume data is already indexed and prepared. The authors argue we need "in-situ" exploration that allows users to start visualizing massive, unknown datasets on the fly without hours of pre-processing.
  • Aesthetics as an Objective Function: We can now use Graph Neural Networks (GNNs) to optimize for "hidden" objective functions—essentially teaching a computer what a "good" or "clear" chart looks like based on human feedback.

Key Results

The paper synthesizes qualitative findings across several sub-fields:

  • Scalability Gap: The authors note that only 2% of articles at the premier IEEE VIS conference focus on scalability, compared to much higher percentages in the database and ML communities.
  • VLM Unlocks: Recent Vision-Language Models have enabled "zero-shot" semantic querying, allowing users to find complex events in video without manual tagging, though at the cost of high computational overhead and potential bias.
  • Layout Optimization: Research shows that GAN-based discriminators can be trained to choose "better" graph layouts by learning from pairs of "good" and "bad" examples, outperforming traditional algorithmic heuristics.

Limitations and Future Work

  • Trust and Hallucination: A major bottleneck is the "brittleness" of AI. If a VLM incorrectly tags a video frame, it can lead a researcher down a false path. Future work must focus on "human-in-the-loop" oversight and fast verification mechanisms (like "highlight reels").
  • The "Unicorn Designer" Problem: Currently, building these systems requires an expert who understands domain knowledge, visualization design, and backend system optimization. The authors suggest we need tools that can automatically generate these end-to-end stacks.
  • Cross-Disciplinary Silos: The biggest hurdle is cultural. The systems, AI, and visualization communities often work in isolation. The paper concludes that "sustained cross-disciplinary collaboration" is the only way to build truly human-centered AI systems.

Summary updated

Abstract

The rapid advancement of AI is transforming human-centered systems, with profound implications for human-AI interaction, human-data interaction, and visual analytics. In the AI era, data analysis increasingly involves large-scale, heterogeneous, and multimodal data that is predominantly unstructured, as well as foundation models such as LLMs and VLMs, which introduce additional uncertainty into analytical processes. These shifts expose persistent challenges for human-data interactive systems, including perceptually misaligned latency, scalability constraints, limitations of existing interaction and exploration paradigms, and growing uncertainty regarding the reliability and interpretability of AI-generated insights. Responding to these challenges requires moving beyond conventional efficiency and scalability metrics, redefining the roles of humans and machines in analytical workflows, and incorporating cognitive, perceptual, and design principles into every level of the human-data interaction stack. This paper investigates the challenges introduced by recent advances in AI and examines how these developments are reshaping the ways users engage with data, while outlining limitations and open research directions for building human-centered AI systems for interactive data analysis in the AI era.
BibTeX
@misc{fekete2026human,
  author = {Jean‐Daniel Fekete and Yifan Hu and Dominik Moritz and Arnab Nandi and Senjuti Basu Roy and Eugene W. Wu and Nikos Bikakis and George Papastefanatos and Panos K. Chrysanthis and Guoliang Li and Lingyun Yu},
  title = {{Human-Data Interaction, Exploration, and Visualization in the AI Era: Challenges and Opportunities}},
  year = {2026},
  eprint = {2603.05542},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2603.05542},
}
Figure