Tools that recommend charts during data exploration usually judge a chart "interesting" when a filtered slice of the data looks very different from the rest. SEEDB is the clearest example: it ranks candidate views by the statistical distance between a chosen subset and a reference population. That rule cannot tell three kinds of large deviation apart. Some are facts every domain expert already knows, such as older people earning more in census data. Some are artifacts of how the data were collected. Only a few are findings worth an analyst's attention. Telling them apart takes knowledge from outside the table.
We propose to rank views against what outside knowledge predicts, not against the rest of the data. Before the system looks at a candidate view, a language model with access to retrieved domain documents writes down a forecast of what that view should show. The view's score is how far the observed data departs from that forecast. The model never sees the data and never judges the finding. It only commits to a prediction, and the data decide. The aim is to bring in outside knowledge while avoiding the habit, well documented in current models, of agreeing with whatever they are shown.
Core Claim
A view's value to an analyst is better measured as surprise relative to a knowledge-informed forecast than as deviation from a reference subset. The forecast must be produced blind, before the data are seen, so that outside knowledge shapes the expectation but cannot override the evidence.
The claim draws on three lines of work. The Rational Agent Benchmark for Data Visualization defines a visualization's value of information as the score gain of an ideal Bayesian agent who sees it, relative to the same agent acting on its prior alone. Applying this framework, it found that the no-visualization baseline can be "surprisingly high." In other words, prior knowledge already explains much of what a chart shows, and that is exactly what deviation-based recommenders ignore. SEEDB supplies the machinery we reuse: candidate view enumeration, shared aggregate queries, and bandit and confidence-interval pruning. We change only its reference distribution.
AssistedDS constrains the design. When language models doing data science were handed domain hints, they followed harmful ones 60–90% of the time, and following them degraded downstream performance by up to 28%. VisJudge-Bench found a related problem: multimodal models inflate their scores for flawed charts. Both results rule out the obvious design of letting a model read the chart and decide whether it is interesting.
The scoring rule follows from the rational-agent view. Let $\hat P_v$ be the observed distribution of view $v$ (for example, mean income per education level within the analyst's filter). Let $Q^K_v$ be the forecast distribution the model produces from retrieved knowledge $K$ without seeing the data. The score is
$$s(v) = \mathrm{KL}\big(\hat P_v \,\|\, Q^K_v\big) - \lambda \, u(v),$$
where $u(v)$ is the sampling uncertainty of $\hat P_v$ (for example, a bootstrap estimate of the divergence's spread) and $\lambda$ is a weight tuned on held-out data. Under a logarithmic scoring rule, the KL term is exactly the expected score an agent holding prior $Q^K_v$ gains by seeing the data. That makes $s(v)$ a computable stand-in for the rational-agent value of information. SEEDB's rule is the special case where $Q$ is the empirical reference distribution and outside knowledge plays no part.
Research Program
The system has four parts.
Grounding the view. We convert each candidate view (filter predicate, grouping attribute, measure, aggregate) into a text query naming the entities involved. Examples are "median house price by county, restricted to coastal counties" or "flight delay by carrier." This adapts entity-centric retrieval, as in EntRAG, from natural images to abstract chart marks. For charts the entity is already explicit in the database schema, which sidesteps the visual recognition bottleneck that INFOSEEK identified.
Forecasting. We retrieve passages from a domain corpus and from the dataset's documentation. The documentation matters because it often describes collection quirks, which lets the forecast anticipate artifacts. The model then outputs a distribution over the view's groups. To stay within an interactive budget, the model forecasts only views that survive SEEDB's pruning. Forecasts are cached per (predicate, attribute, measure) and are coarsened to group rankings with confidence when time is short.
Scoring. The data are compared with the forecast using the equation above.
Explanation. The interface shows each recommended view next to its forecast and the retrieved passages behind that forecast. The analyst sees why the view was flagged, which addresses the low explainability and controllability scores that the dual expert/end-user guidance heuristics recorded for recommenders like Voyager.
The analysis question is when blind forecasting helps. We expect two answers. It should help when the knowledge corpus covers the obvious effects in the data, because those views lose their score. It should also help when the corpus contains misleading claims, because a false forecast then makes the true data look surprising, which flags the conflict instead of hiding it.
Baselines
Plain SEEDB. The naive reference point is SEEDB with Earth Mover's Distance against the complement subset, run on the same candidate views and the same pruning. Its reported result is an AUROC of 0.903 against expert interestingness labels on Census (SEEDB). We reproduce that number first. We expect our method to roughly match it on generic interestingness, and to beat it on the narrower target of views that are interesting and not already known.
Significance-filtered SEEDB. This is SEEDB plus permutation tests with Benjamini–Hochberg false-discovery control. It is the strongest knowledge-free way to reject spurious deviations, and it shares our uncertainty term $u(v)$, so the comparison isolates the effect of outside knowledge.
Post-hoc language-model reranker. This is the strongest competitor. It uses the same model, the same retrieved passages, and the same candidate views. The model sees SEEDB's top views, including their numbers, and rates each as known, artifact, or novel. Prompt-tuning gets the same budget of held-out examples as our $\lambda$ tuning. No reported result was found on Lacuna for this baseline on interactive view recommendation. The closest evidence is that GPT-5 reaches only 0.428 correlation with human chart-quality judgments (VisJudge-Bench). Our method beats this baseline if it ranks better under misleading knowledge while staying within noise under clean knowledge.
Ablations. Two ablations isolate the key ingredients. The first lets the forecaster see the data, which removes blindness. The second drops retrieval so the model forecasts from its own weights alone.
Why not just…?
Why not just ask a language model to rerank SEEDB's output? This is a real threat. With accurate documents, a post-hoc reranker may recognize known facts as well as blind forecasting does. The difference should show up when the documents are wrong or when a striking chart invites agreement. AssistedDS found adoption of harmful hints at 60–90%, and VisJudge-Bench found inflated scores for flawed charts. Our misleading-knowledge condition tests this directly. If the reranker matches us there too, blindness is unnecessary and we will report that the simpler design suffices.
Why not just use better statistics? Multiple-testing correction removes deviations that are noise. It keeps deviations that are real but already known, and it keeps systematic collection artifacts, because those are statistically solid. The significance-filtered baseline measures how much of the gain needs no outside knowledge. Our synthetic data separate "real but known" and "artifact" cases on purpose.
Why not just show the analyst the retrieved documents and let them judge? Humans also anchor on prior beliefs. How to evaluate data visualizations across different levels of understanding found that viewers reached the same policy conclusions whether a chart was misleading or redesigned. Leaving the comparison to the user moves the problem rather than solving it. The user study below includes this "documents alongside SEEDB" condition.
Proposed Experiments
Controlled ground truth. We use the AssistedDS synthetic datasets, whose generating functions are known. In each dataset we plant three kinds of effect: documented effects (written into the knowledge corpus), undocumented effects (true novel findings), and collection artifacts (for example, one site's duplicated rows, either mentioned or not in the documentation). We run four knowledge conditions mirroring AssistedDS: helpful only, adversarial only, mixed, and misleading. The primary metric is ranking quality on the undocumented-effect label: precision at 5 and AUROC. Secondary metrics are how often adversarial claims suppress a true effect and how often conflicting views are flagged.
Real data. On SEEDB's Census, Housing, and Bank datasets, domain annotators label top deviation views as known, artifact, or novel. We report AUROC against both the original interestingness labels and the novelty labels.
Latency. We use the IDEBench flights workloads (IDEBench) and report the fraction of interactions that violate the 1s and 3s thresholds once forecasting is added.
User study. Participants explore a dataset and then make incentivized predictions about held-out slices. Following the rational-agent framework, we compute each condition's realized value of information and split errors into belief loss (misreading the data) and optimization loss (failing to act well on correct beliefs).
Decisive check. The data-visible ablation should match our method under clean knowledge and degrade under misleading knowledge. If it does not degrade, the central mechanism is unsupported.
References
- The Rational Agent Benchmark for Data Visualization
- SEEDB: Efficient Data-Driven Visualization Recommendations to Support Visual Analytics
- AssistedDS: Benchmarking How External Domain Knowledge Assists LLMs in Automated Data Science
- VisJudge-Bench: Aesthetics and Quality Assessment of Visualizations
- EntRAG: Entity-Centric Retrieval-Augmented Generation for Knowledge-based Visual Question Answering
- Can Pre-trained Vision and Language Models Answer Visual Information-Seeking Questions?
- A Heuristic Approach for Dual Expert/End-User Evaluation of Guidance in Visual Analytics
- How to evaluate data visualizations across different levels of understanding
- IDEBench: A Benchmark for Interactive Data Exploration