Pay "Attention" to Chart Images for What You Read on Text

Chenyu Yang, Ruixue Fan, Nan Tang, Meihui Zhang, Xiaoman Zhao, Ju Fan, Xiaoyong Du

Abstract

Data visualization is changing how we understand data, by showing why's, how's, and what's behind important patterns/trends in almost every corner of the world, such as in academic papers, news articles, financial reports, etc. However, along with the increasing complexity and richness of data visualizations, given a text description (e.g., "fewer teens say they attended school completely online (8%)"), it becomes harder for users to pinpoint where to pay attention to on a chart (e.g., a grouped bar chart). In this demonstration paper, we present a system HiChart for text-chart image highlighting: when a user selects a span of text, HiChart automatically analyzes the chart image (e.g., a jpeg or a png file) and highlights the parts that are relevant to the span. From a technical perspective, HiChart devises the following techniques. Reverse-engineering visualizations: given a chart image, HiChart uses computer vision techniques to generate a visualization specification using Vega-Lite language, as well as the underlying dataset; Visualization calibration by data tuning: HiChart calibrates the re-generated chart by tuning the recovered dataset through value perturbation; and Chart highlighting for a span: HiChart maps the span to corresponding data cells and uses the built-in highlighting functions of Vega-Lite to highlight the chart.
BibTeX
@inproceedings{yang2023pay,
  author = {Chenyu Yang and Ruixue Fan and Nan Tang and Meihui Zhang and Xiaoman Zhao and Ju Fan and Xiaoyong Du},
  title = {{Pay "Attention" to Chart Images for What You Read on Text}},
  booktitle = {SIGMOD Conference Companion},
  year = {2023},
  doi = {10.1145/3555041.3589714},
}

Loading related content...

Figure