AI2D-RST: A multimodal corpus of 1000 primary school science diagrams

Tuomo Hiippala, Malihe Alikhani, Jonas Haverinen, Timo Kalliokoski, Evanfiya Logacheva, Serafina Orekhova, Aino Tuomainen, Matthew Stone, John A. BatemanarXiv 2019 · DOI · arXiv

TL;DR

AI2D-RST introduces a multi-layered, graph-based corpus of 1,000 primary school science diagrams annotated using Rhetorical Structure Theory (RST). By decoupling explicit visual layouts from implicit discourse relations, this dataset provides a rich, structurally sound framework for training machine learning models to truly "read" and reason about complex multimodal graphics.

What Problem Does This Paper Address?

Diagrams are essential tools for communicating scientific concepts, yet teaching computers to understand them remains a massive challenge. Most computer vision and natural language processing models treat diagram parsing as a simple object detection or localized matching problem (e.g., mapping a text label to an arrow).

However, diagrams operate as a form of discourse. The meaning of an arrow changes entirely depending on whether it appears in a food web, a lifecycle, or a circuit diagram. Existing datasets fail to capture this global organization and the implicit relationships between groups of elements. Without understanding how visual entities cluster into perceptual units and how those units functionally relate to one another, AI models struggle with high-level tasks like Visual Question Answering (VQA).

Methods

The authors address this gap by developing a multi-layer annotation schema where each layer is represented as a graph. Instead of flattening a diagram's structure, they break it down into four distinct, interconnected layers:

  1. Grouping: Clusters individual diagram elements (text, blobs, arrows) into perceptual units based loosely on Gestalt principles of perception. This is represented as an undirected, acyclic tree graph.
  2. Macro-grouping: Captures the overarching structural category or "type" of the diagram (e.g., a cycle, a network, a cross-section, or a table).
  3. Connectivity: Maps out visually explicit connections signaled by arrows or lines, tracking clear sources and targets via a cyclic mixed graph.
  4. Discourse Structure: Applies Rhetorical Structure Theory (RST) to define the functional, semantic relationships (such as elaboration, preparation, or joint) between elements and groups. This is modeled as a directed acyclic tree graph.

To build the corpus, five trained expert annotators used an in-house command-line Python tool to systematically construct these graphs on top of pre-existing layout segmentations.

Figure 1: (1) A thumbnail of the original diagram image scraped from the web, (2) its crowd-sourced layout segmentation (converted into greyscale to bring out the annotation) and (3) a Diagram Parse Graph (DPG) for diagram #274 in AI2D. Diagram element types are coded using same colours in both layout segmentation and DPG: text blocks (blue), blobs (red), arrows (green), arrowheads (orange) and image constant (Navajo white).

Datasets

The newly introduced dataset is AI2D-RST. It consists of 1,000 English-language natural science diagrams representing primary school topics (ages 6–11), such as human physiology, moon phases, and food webs.

AI2D-RST is built directly on top of a subset of the Allen Institute for Artificial Intelligence Diagrams (AI2D) dataset, which contains 4,903 diagrams with crowd-sourced layout segmentations. AI2D-RST inherits the unique identifiers from AI2D, allowing for clean cross-referencing across the new annotation layers.

Key Insights and Ideas

  • Diagrams as Text: A core insight is that understanding a diagram is a discourse-level problem. Just like sentences in an essay, diagrammatic elements act as "elementary discourse units" that rely on context to deliver a cohesive message.
  • The "Reuse" of Elements: Diagrams frequently reuse a single label or image across multiple discourse relations. To preserve a clean hierarchical tree structure for RST analysis, the authors introduced a "node splitting" technique (e.g., duplicating a text node into T7.1 and T7.2) so an element can simultaneously serve different rhetorical roles (like being a satellite to two different nuclei).
  • Schematic Resistance to Discourse: The authors discovered that highly standardized schematic diagrams (like electrical circuits) resist traditional RST discourse analysis. Because circuit symbols have fixed, non-contextual meanings, the viewer does not need to resolve implicit discourse relations—all necessary information is explicitly carried by the connectivity layer.

Key Results

  • High Annotator Reliability: The expert annotations achieved strong inter-annotator agreement across all layers, measured using Fleiss' kappa ($\kappa$). Grouping achieved a marginal $\kappa$ of 0.836, Connectivity reached 0.878, Macro-grouping scored 0.784, and the highly abstract Discourse Structure layer maintained a promising $\kappa$ of 0.733.
  • Structural Clustering Validation: To explore the corpus, the authors generated 46-dimensional feature vectors for each diagram based on element counts, relation types, and network density. When visualized using a 2D UMAP embedding, the diagrams formed distinct, clear clusters corresponding to their macro-groups (e.g., cycles vs. cut-outs). This proved that simple structural and rhetorical counts successfully capture high-level diagrammatic styles and layouts.

Limitations and Future Work

  • Granularity Mismatch: Because AI2D-RST relies on AI2D's original crowd-sourced layout segmentations, the annotators could not decompose individual visual elements further. For instance, if a diagram uses labels to point to specific parts of a single animal illustration, a perfect RST analysis would require decomposing that illustration into its component parts—a level of detail the base dataset lacks.
  • Expert Bias and Circularity: The annotation schema was refined collaboratively among the expert annotators. While this boosted agreement, the authors note it introduces implicit, shared knowledge that could limit reproducibility among naive annotators.
  • Scaling via Automation: The 1,000 annotated diagrams represent just over 20% of the full AI2D dataset. The authors propose using this new corpus as a training benchmark for graph neural networks to automatically predict and generate AI2D-RST representations for the remaining 3,900 diagrams.

Summary updated

Abstract

This article introduces AI2D-RST, a multimodal corpus of 1000 English-language diagrams that represent topics in primary school natural sciences, such as food webs, life cycles, moon phases and human physiology. The corpus is based on the Allen Institute for Artificial Intelligence Diagrams (AI2D) dataset, a collection of diagrams with crowd-sourced descriptions, which was originally developed to support research on automatic diagram understanding and visual question answering. Building on the segmentation of diagram layouts in AI2D, the AI2D-RST corpus presents a new multi-layer annotation schema that provides a rich description of their multimodal structure. Annotated by trained experts, the layers describe (1) the grouping of diagram elements into perceptual units, (2) the connections set up by diagrammatic elements such as arrows and lines, and (3) the discourse relations between diagram elements, which are described using Rhetorical Structure Theory (RST). Each annotation layer in AI2D-RST is represented using a graph. The corpus is freely available for research and teaching.
BibTeX
@misc{hiippala2019ai,
  author = {Tuomo Hiippala and Malihe Alikhani and Jonas Haverinen and Timo Kalliokoski and Evanfiya Logacheva and Serafina Orekhova and Aino Tuomainen and Matthew Stone and John A. Bateman},
  title = {{AI2D-RST: A multimodal corpus of 1000 primary school science diagrams}},
  year = {2019},
  eprint = {1912.03879},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1912.03879},
}
Figure