Learning When Removed Chart Elements Need Replacing

When a chart designed for a desktop screen is squeezed onto a phone, something has to go: marks, labels, gridlines, an axis, a legend. Designers routinely cut detail and then compensate, for example by adding a text callout where an axis used to carry the comparison. Today's automated tools cannot decide when such a swap is warranted. They score whole candidate charts with fixed weights, so they have no notion that a label's value depends on what else is still on the screen. We propose to model a chart as a set of elements whose joint contribution to the chart's message shows diminishing returns. We learn that set-scoring function so that its weights shift with context: viewport size, how crowded the surviving marks are, and the reader's task. Once the function is learned, adapting a chart becomes budgeted set selection. The compensate action then falls out of the structure of the function instead of being hand-written as a rule. An annotation and the axis it replaces act as substitutes, and substitutes are exactly what a diminishing-returns (submodular) function represents.

In two sentences: responsive visualization systems currently judge "how much of the message survived" with one global scorer over whole charts. We replace that scorer with a context-conditioned, element-level submodular value, so the system can decide which specific elements to keep, drop, or swap under a screen budget, and why.

Where the two directions meet, and what is already done

The density-message trade-off is the observation that a display's usefulness rises with detail up to a point and then falls as the reader is overloaded. It was named and catalogued for responsive charts by Design Patterns and Trade-Offs in Responsive Visualization for Communication. That work coded 378 desktop/mobile pairs from outlets such as the New York Times and found "Rescale" and "Remove" to be the dominant actions. It left open how to formalize the intended message, and this paper is the source of our motivation. The same curve appears outside charts: Number and quality of diagrams in scholarly publications is associated with number of citations links one or two good diagrams, not more, to higher citations, and Visualizing indoor layout for spatial learning using Mixed Reality-based X-ray vision reports fatigue from excess visibility. Both motivate treating detail as having a context-dependent, eventually negative, marginal value.

The closest existing system is An Automated Approach to Reasoning About Task-Oriented Insights in Responsive Visualization. It generates candidate mobile views with an answer-set-programming search and scores each with three hand-designed losses: entropy for identification, Earth Mover's Distance for comparison, and a LOESS area difference for trends. A Random Forest trained on nine experts' pairwise rankings of 2,120 alternatives then predicts which view experts prefer, at 84% accuracy. Dupo: A Mixed-Initiative Authoring Tool for Responsive Visualization wraps that ranking in an authoring tool where users "lock" elements by hand. Cicero: A Declarative Grammar for Responsive Visualization supplies the transformation vocabulary (modify, transpose, remove, compensate) but leaves the choice of which rule to apply to the human. All three supply method and infrastructure that we reuse, namely candidate generation, losses, and the action grammar. None of them scores individual elements, and none represents the interaction between a removed encoding and its replacement.

On the weighting side, Learning Interpretable Feature Context Effects in Discrete Choice lets feature weights shift linearly with the average features of the choice set. Preference Learning of Latent Decision Utilities with a Human-like Model of Preferential Choice models decoy and cross-feature effects with neural surrogates, but only over fixed-size choice sets. Both show that fixed weights misdescribe human judgment. Neither has been applied to visual elements, and the linearity of the first and the fixed set size of the second both constrain our design. Improving Attribution Methods by Learning Submodular Functions trains a deep submodular function, a small network with non-negative weights and concave activations that is guaranteed to show diminishing returns. It uses that function to value an image feature by its marginal gain given features already chosen, and it has only been used for classifier heatmaps. We borrow its function class and make it context-conditioned. We also change its role from explaining a classifier to allocating a screen budget.

The obvious combination would be to feed context features into Kim et al.'s Random Forest. That is not our proposal, and we treat it as a baseline. A Random Forest over whole-view losses can learn context interactions, but it only ranks views it is handed. It cannot say which element to add back, and it has no representation of substitution between elements.

Core Claim

We claim that the message a chart carries is well modeled by a monotone submodular value over its elements whose weights depend on context. We further claim that maximizing this value under a screen-area budget reproduces expert and professional adaptation decisions, including when to compensate, better than whole-view scorers. Concretely, let $V$ be the elements of the source chart (mark groups, axes, labels, legend entries, and a pool of candidate annotations). Let $c$ be a context vector: viewport dimensions, the density of marks currently retained, and a task descriptor. Let $\mathrm{cost}(e)$ be the pixel area that element $e$ occupies at the target size. We learn

$$ f_\theta(S; c) = \sum_{k} w_k(c)\, \phi_k\Big(\sum_{e \in S} m_{ke}(c)\Big), \qquad S^\star = \arg\max_{S \subseteq V,\ \sum_{e\in S}\mathrm{cost}(e) \le B(c)} f_\theta(S; c). $$

Here $S$ is a subset of elements, $\phi_k$ are fixed concave functions, and $w_k(c) \ge 0$ and $m_{ke}(c) \ge 0$ are produced from the context by a small network with a softplus output. Because the weights stay non-negative and the $\phi_k$ stay concave, $f_\theta$ is submodular for every context. Cost-benefit greedy selection therefore keeps its standard constant-factor approximation guarantee for budgeted submodular maximization. The context network is what makes an axis matter more for a comparison task than for a trend task, and what makes an extra label worth less in a crowded region than in a sparse one.

Research Program

The first component is data at the level of elements. We will parse the Cicero use cases and a sample of the professional corpus from the design-patterns study into element sets with desktop-to-mobile correspondences. Cicero's specifications already mark which elements were removed and which were compensated, which gives direct supervision for the substitution question. The expert preference dataset from Kim et al. provides pairwise judgments between whole views, and each view is a subset of $V$. The existing labels therefore train $f_\theta$ through a Bradley-Terry likelihood on $f_\theta(S_a;c) - f_\theta(S_b;c)$. Professional mobile designs are treated as revealed choices: the chosen subset should beat sampled feasible subsets of equal cost. This yields a set-choice likelihood that, unlike the surrogates of De Peuter et al., handles choice sets of any size.

The second component is a crowdsourced perception study that grounds "message" in reader behavior instead of expert taste alone. Participants see adapted charts and answer the source chart's intended takeaway question (identify, compare, or trend, matching the three task families of Kim et al.). Takeaway accuracy becomes an additional training target and the main outcome measure. The third component is the adaptation engine itself. Greedy selection over $f_\theta$ is compiled into Cicero actions, so outputs are ordinary responsive specifications a designer can edit.

Baselines

The naive reference points are keeping every element and rescaling uniformly, which is what standard responsive CSS does, and the simple rules evaluated by Kim et al. Those rules scored 59% (size-based heuristics) and 63% (axis-transposition rules) on expert preference prediction in An Automated Approach to Reasoning About Task-Oriented Insights in Responsive Visualization. Any learned method must clear these by a wide margin.

The strongest existing method is that paper's Random Forest over insight losses, reported at 84% accuracy on expert pairwise preferences. We will retrain it on our splits with the same features, so both methods see identical data and an equal hyperparameter search budget. We will also build a stronger version that adds our context vector and element counts as inputs, tuned as gradient-boosted trees. On whole-view preference prediction, our model must match this context-augmented scorer within confidence intervals. Beating it there is not the main claim. It must beat the scorer on the two things a whole-view scorer cannot do directly: choosing designs under a budget, and deciding when to compensate. To make that comparison fair, the tree scorer is given the same candidate generator. We then use it to pick the best view among $N$ sampled budget-feasible subsets, with $N$ set so that wall-clock time matches greedy selection over $f_\theta$. Dupo's ASP-ranked recommendations serve as the deployed-system reference. For Dupo, the file reports only authoring-time results (3.0 versus 5.3 minutes) and no accuracy figure.

Three ablations isolate the key ingredients. A modular model with the same context network but additive element values tests whether diminishing returns matters; it cannot represent substitution. A context-free submodular model with fixed $w, m$ tests whether context matters. A linear context version, in which weights shift linearly with the mean element features in the style of the Linear Context Logit, tests whether non-linear context dependence matters. Beating each ablation on held-out charts, especially on compensation decisions, is what would establish each ingredient.

Why not just…?

Why not just add context features to the Random Forest? This is the most serious threat for preference prediction, and we expect it to be competitive there. The experiment above is designed to show where it stops. A whole-view scorer ranks views only after something else has generated them, so its budget-constrained quality depends on sampling many subsets. It also gives no element-level account of why a callout was added. If the tree scorer given matched search time equals our model on takeaway accuracy and compensation recall, the element-level structure has not earned its place. We would then report the contribution as the element-level dataset and analysis rather than the model.

Why not just score elements independently and delete the lowest-value ones first? The modular ablation is exactly this. It will keep an axis and a redundant annotation together because each looks valuable alone. It will also fail to add an annotation after the axis is dropped, because an element's value cannot change with what else is on screen. Compensation recall on the Cicero cases directly measures this failure.

Why not just ask a vision-language model to adapt the chart? This approach is cheap and deserves a direct test. The only evidence in the file is indirect. In Presenting a Paper is an Art: Self-Improvement Aesthetic Agents for Academic Presentations, a specialized model reached 87.8% on slide aesthetic pairwise comparisons, outperforming GPT-4o by "nearly 40%". I found no reported result on Lacuna for vision-language models on responsive visualization. We will prompt GPT-4o with the source chart, its specification, the target size, and the takeaway question, and we will score its edited specifications with the same takeaway-accuracy study.

Proposed Experiments

The first experiment is whole-view preference prediction on the Kim et al. expert dataset, split by source visualization so that each test chart is unseen. The metric is pairwise accuracy against the 84% Random Forest and its context-augmented version. The second is reproduction of professional designs on the parsed corpus and Cicero cases. We measure Jaccard overlap between the predicted and the designer's retained element sets, and precision and recall of compensate actions. The substitution test sits inside this experiment: in cases where a designer removed an encoding and added text, our model should add the annotation only when the corresponding axis or mark is absent. The third and decisive experiment is the crowd study. Adapted charts from every method are shown at three shrinking budgets, and we measure takeaway accuracy and response time. The prediction is that our method's advantage grows as the budget tightens, because that is where context-dependent weights and substitution matter most.

A final check probes the overload side of the density-message curve. We add a small density cost to $f_\theta$ and ask whether the learned value produces an interior optimum, with fewer elements preferred even when space allows more, in a direction consistent with the diminishing-returns patterns the diagram-citation study reports. If our model fails the substitution test or loses on takeaway accuracy to the context-augmented trees, the central claim is wrong.