Kori: Interactive Synthesis of Text and Charts in Data Documents

Shahid Latif, Zheng Zhou, Yoon Kim, Fabian Beck, Nam Wook Kim

TL;DR

This paper introduces Kori, a novel mixed-initiative interface that helps authors create interactive data documents by seamlessly linking text and charts. Kori leverages natural language processing to automatically suggest these links and provides intuitive tools for manual creation, addressing the cognitive burden of switching between text and visuals.

What Problem Does This Paper Address?

Communicating complex data effectively often relies on a combination of text and charts. While charts offer visual summaries, text provides context and explanation. However, synthesizing information across these two distinct modalities is challenging for readers, forcing them to constantly shift their attention. This "split-attention effect" can hinder comprehension and learning. Existing tools for data storytelling often treat text and charts as separate entities, requiring advanced programming skills to create interactive links between them. This paper aims to bridge this gap by providing an accessible authoring tool that facilitates the tight coupling of text and charts.

Methods

Kori employs a mixed-initiative approach, combining automatic suggestions with manual creation tools.

  • Automatic Reference Detection: Kori uses natural language processing (NLP) to analyze user-typed text and identify potential references to charts. This involves:
  • Chart Feature Extraction: Extracting key information from charts, such as axis labels, values, legends, and data properties.
  • Point-level Matching: Using vector representations (from FastText) to match text phrases to chart features, accounting for typos and variations.
  • Interval-level Matching: Employing rule-based pattern matching on part-of-speech tags to identify numerical intervals mentioned in text.
  • Reference Grouping: Combining identified point and interval references into higher-level references, mimicking the hierarchical structure of sentences.
  • Manual Reference Construction: Kori offers two modes for manual linking:
  • Direct Manipulation: Users can select text and then directly "brush" or select corresponding visual marks on the chart. This mode supports "relaxation" of selections, allowing users to easily select multiple similar marks.
  • Filtering Mode: Users can set filters for data dimensions to define references, offering more precise control, especially for complex charts or when direct manipulation is difficult.

Datasets

The authors analyzed a collection of text-chart pairs from news articles and scientific papers. This collection was expanded from a previous dataset by Kong et al. [28] to include more chart types and venues. For algorithmic evaluation, a new dataset was curated, consisting of 92 chart-text pairs with 15 different chart types, including bar charts, scatter plots, and heatmaps. This dataset was split into validation and test sets.

Key Insights and Ideas

  • References as Selections: A core insight is that text-chart references fundamentally mirror selection operations within visualizations (e.g., selecting a point, a range, or multiple points).
  • Hierarchical Grouping: Text-chart references are not just isolated links but can be hierarchically grouped, forming a structure similar to a sentence's parse tree. This allows for more complex and nuanced connections.
  • Data Transformations: The grouping of references can implicitly represent data transformations, such as unions (combining multiple countries) or intersections (combining ranges of values).
  • Exploration as a Byproduct: The authoring interface, designed for creating links, also naturally supports data exploration, as users can use the linking process to verify statements or discover insights.

Key Results

  • Algorithmic Evaluation: The automatic reference detection pipeline achieved an F₁ score of 0.34 on the validation dataset. While point reference detection performed reasonably well (F₁ = 0.47), interval detection and reference grouping were more challenging.
  • User Study: 11 participants, with diverse backgrounds, successfully completed authoring tasks. They found Kori intuitive, easy to use, and highly useful (median rating of 5/5 for usefulness). Participants rated the manual reference construction interface higher than the automatic suggestions.
  • Usage Patterns: Participants frequently used direct manipulation for maps and line charts, and the filtering mode for scatter plots and bubble charts.

Limitations and Future Work

  • Ambiguity in Suggestions: Kori's automatic suggestions can sometimes be ambiguous, especially when charts share underlying datasets. Future work could explore more sophisticated methods to resolve this ambiguity, potentially by considering user context or cursor position.
  • One-to-One Mapping: Currently, Kori primarily supports one-to-one mapping between text and charts. Exploring one-to-many or many-to-many mapping models could offer more flexibility but might complicate the interface.
  • Improving Automatic Linking: Further research is needed to improve the reliability and accuracy of the automatic suggestion pipeline, possibly by exploring end-to-end models trained on larger datasets and leveraging advanced NLP techniques like BERT.
  • Supporting Exploration: The authors suggest extending Kori to better support the full lifecycle of data-driven storytelling, including data exploration and chart creation, rather than assuming pre-existing charts.
  • Beyond Text-Chart: Future work could explore adapting Kori's approach to other forms of data stories, such as data comics, and investigating alternative methods for reducing cognitive load, like spatial integration of text and graphics.

Summary updated

Abstract

Charts go hand in hand with text to communicate complex data and are widely adopted in news articles, online blogs, and academic papers. They provide graphical summaries of the data, while text explains the message and context. However, synthesizing information across text and charts is difficult; it requires readers to frequently shift their attention. We investigated ways to support the tight coupling of text and charts in data documents. To understand their interplay, we analyzed the design space of chart-text references through news articles and scientific papers. Informed by the analysis, we developed a mixed-initiative interface enabling users to construct interactive references between text and charts. It leverages natural language processing to automatically suggest references as well as allows users to manually construct other references effortlessly. A user study complemented with algorithmic evaluation of the system suggests that the interface provides an effective way to compose interactive data documents.
BibTeX
@article{latif2022kori,
  author = {Shahid Latif and Zheng Zhou and Yoon Kim and Fabian Beck and Nam Wook Kim},
  title = {{Kori: Interactive Synthesis of Text and Charts in Data Documents}},
  journal = {IEEE Trans. Vis. Comput. Graph.},
  year = {2022},
  url = {https://openreview.net/forum?id=PB5ba0dtThy},
}
Figure