Detecting and Dismantling Composite Visualizations in the Scientific Literature

Po-Shen Lee, Bill Howe

Abstract

In the scientific literature, a single figure often contains a complex grid of multiple charts, labels, and photographs. These "composite visualizations" account for approximately 35% of all figures but present a significant challenge for automated data extraction and academic search engines. This paper presents an automated pipeline designed to "dismantle" these complex images into their individual components by combining recursive splitting with machine learning-based merging.

Introduction

For human readers, parsing a figure containing a bar chart, an electrophoresis gel, and a legend is intuitive. For machines, however, these composite visualizations are often treated as "black boxes." Standard image segmentation frequently fails because it cannot distinguish between a meaningful sub-figure and "auxiliary" fragments, such as floating legends or axis labels. This difficulty hinders the indexing of visual data and the large-scale analysis of visual communication in science (viziometrics). To address this, we propose a three-step dismantling algorithm that leverages the inherent hierarchical structure of scientific layouts.

Methodology

The dismantling process is divided into three primary stages: recursive splitting, SVM-based merging, and scoring.

1. Recursive Splitting

The algorithm first searches for "fire lanes"—blank horizontal or vertical rows or columns of pixels that act as buffers between sub-figures. By utilizing pixel-value histograms and variance thresholds, the system recursively breaks the image into smaller and smaller blocks based on these whitespace guides.

Identification of 'fire lanes' using blank coverage histograms to guide the recursive splitting of composite images.

2. SVM-Based Merging

Recursive splitting often results in fragments (like a title or legend) rather than standalone charts. To resolve this, we trained a Support Vector Machine (SVM) classifier using 15 geometric and whitespace features (e.g., aspect ratio, "blank coverage"). The SVM distinguishes between "standalone" sub-figures and "auxiliary" fragments, allowing the system to merge fragments into their nearest standalone neighbor.

The dismantling process: recursive splitting followed by SVM-based merging to reconstruct coherent sub-figures.

3. Scoring and Selection

Because an image can be split in multiple orders (horizontal vs. vertical first), the system generates various possible decomposition "trees." A custom scoring function that rewards visual balance and repeating patterns is used to select the most likely correct decomposition.

Datasets and Evaluation

The system was trained and evaluated using figures randomly extracted from the PubMed database. The corpus included 1,947 figures for detection training and 213 composite figures (split into 7,541 sub-images) for classifier training. Evaluation was conducted on a set of 261 multi-chart figures containing 1,534 individual visualizations.

Key Results

  • Detection Accuracy: The system correctly identified composite vs. single-chart figures with 90.2% accuracy.
  • Dismantling Performance: For common cases (eight or fewer sub-figures), the algorithm achieved 80.1% recall and 85.1% precision for perfect extractions.
  • Visualization Diversity: The system was 92.3% accurate at identifying the types of visualizations present, even when the extraction was not pixel-perfect.
  • Complexity Constraints: Performance dropped to 42.1% recall on "densely packed" figures containing more than nine sub-figures.

Discussion and Future Work

The current algorithm is optimized for grid-aligned layouts. It struggles with irregular "zigzag" boundaries or "homogeneous arrays" (e.g., a grid of 20 nearly identical microscope photos) where the distinction between separate data points and a single observation is ambiguous. Future research will focus on integrating Optical Character Recognition (OCR) to use text location as a layout hint and incorporating color information to better distinguish between adjacent but distinct plots.

Summary updated

Abstract

We are analyzing the visualizations in the scientific literature to enhance search services, detect plagiarism, and study bibliometrics. An immediate problem is the ubiquitous use of multi-part figures: single images with multiple embedded sub-visualizations. Such figures account for approximately 35 % of the figures in the scientific literature. Conventional image segmentation techniques and other existing approaches have been shown to be ineffective for parsing visualizations. We propose an algorithm to automatically recognize multi-chart visualizations and segment them into a set of single-chart visualizations, thereby enabling downstream analysis. Our approach first splits an image into fragments based on background color and layout patterns. An SVM-based binary classifier then distinguishes complete charts from auxiliary fragments such as labels, ticks, and legends, achieving an average 98.1 % accuracy. Next, we recursively merge fragments to reconstruct complete visualizations. Finally, a scoring function is used to choose between alternative merge trees. For the multi-chart figure detection, we utilize the output of the splitting algorithm as image features to train a classifier. It can avoid unnecessary time consuming by applying the complete algorithm to determine a multi-chart visualization. To evaluate our approach, we randomly collected 880 single-chart scientific figures and 1067 multi-chart scientific figures from the PubMed database. For the detection, we achieve 90.2 % accuracy via 10-fold cross-validation on the entire corpus. To evaluate the decomposition algorithm, we randomly extracted 261 multi-chart figures as a testing set. Our algorithm achieves 80 % recall and 85 % precision of perfect extractions for the common case of eight or fewer sub-figures per figure. Further, even imperfect extractions are shown to be sufficient for most chart classification and reasoning tasks associated with bibliometrics and academic search applications.
BibTeX
@inproceedings{lee2015detecting,
  author = {Po-Shen Lee and Bill Howe},
  title = {{Detecting and Dismantling Composite Visualizations in the Scientific Literature}},
  booktitle = {ICPRAM (Selected Papers)},
  year = {2015},
  doi = {10.1007/978-3-319-27677-9\_16},
}
Figure