A Diagram Is Worth A Dozen Images

Aniruddha Kembhavi, Mike Salvato, Eric Kolve, Min Joon Seo, Hannaneh Hajishirzi, Ali Farhadi

Introduction

While computer vision has achieved remarkable success in natural image understanding, scientific diagrams remain an under-explored frontier. Unlike natural photographs, diagrams are abstract, purposefully designed illustrations that convey complex phenomena—such as the water cycle, food webs, or anatomical structures—through higher-order relationships. These visuals suppress background noise but introduce unique semantic challenges: elements like arrows can represent "consumption," "evaporation," or "flow" depending on context.

Examples of scientific diagrams from the AI2D dataset, illustrating complex relationships like cycles and food webs.

This paper addresses the gap by defining two primary tasks: syntactic parsing (identifying structural constituents) and semantic interpretation (reasoning over these elements to answer questions).

Methodology: From Pixels to Parse Graphs

The authors propose a structured approach to diagram understanding centered on the Diagram Parse Graph (DPG). In a DPG, nodes represent constituents such as text boxes, arrows, and illustrations ("blobs"), while edges represent semantic relationships like labels or linkages.

Syntactic Parsing with DSDP-NET

Standard object detectors often fail on the abstract nature of diagrams. To overcome this, the authors utilize a combination of Random Forests and specialized CNNs to generate candidate nodes and edges. To refine these noisy proposals into a coherent graph, they introduce DSDP-NET (Deep Sequential Diagram Parser).

The DSDP-NET architecture, which uses a stacked LSTM to sequentially build a Diagram Parse Graph (DPG) by evaluating candidate relationships.

DSDP-NET employs a stacked LSTM network to evaluate candidate relationships sequentially. This approach allows the model to incorporate global context—such as spatial redundancy and canvas coverage—rather than relying on local, greedy decisions.

Semantic Reasoning with DQA-NET

To evaluate semantic understanding, the authors developed DQA-NET (Diagram Question Answering). This model frames reasoning as a multiple-choice task. It uses LSTMs to embed both the textual question/answers and the extracted DPG relations. An attention mechanism then identifies the most relevant diagrammatic "facts" to determine the correct answer.

The AI2D Dataset

To facilitate research, the authors released AI2 Diagrams (AI2D), a dataset of over 5,000 grade-school science diagrams. AI2D includes exhaustive crowdsourced annotations:

  • 118,000+ constituents
  • 53,000+ relationships
  • 15,000+ multiple-choice questions

Unlike standard VQA datasets, AI2D requires deep reasoning over scientific concepts and explicit structural relationships.

Results and Insights

The study yields several key findings:

  1. Specialization is Necessary: Standard natural image baselines like Edge Boxes achieved nearly 0% Average Precision (AP) for constituent detection, whereas the authors' custom Random Forest approach reached 0.7829 AP.
  2. Sequential Parsing Wins: DSDP-NET achieved a Jaccard Index for Graphs (JIG) of 51.45, significantly outperforming A* search (41.02) and greedy methods (28.96).
  3. Structure Enables Reasoning: DQA-NET achieved 38.47% accuracy on the AI2D test set, outperforming a standard VQA baseline (32.90%) that treats diagrams as simple pixel grids.

Conclusion

While the absolute accuracy for diagram reasoning remains a challenge, this work demonstrates that explicit structural parsing is vital. Future progress likely depends on integrating external commonsense knowledge to interpret the abstract relationships that diagrams so effectively summarize.

Summary updated

BibTeX
@inproceedings{kembhavi2016diagram,
  author = {Aniruddha Kembhavi and Mike Salvato and Eric Kolve and Min Joon Seo and Hannaneh Hajishirzi and Ali Farhadi},
  title = {{A Diagram Is Worth A Dozen Images}},
  booktitle = {European Conference on Computer Vision},
  year = {2016},
  url = {https://openreview.net/forum?id=WsESld-n6iC},
}
Figure