Abstract
In the scientific literature, a single figure often contains a complex grid of multiple charts, labels, and photographs. These "composite visualizations" account for approximately 35% of all figures but present a significant challenge for automated data extraction and academic search engines. This paper presents an automated pipeline designed to "dismantle" these complex images into their individual components by combining recursive splitting with machine learning-based merging.
Introduction
For human readers, parsing a figure containing a bar chart, an electrophoresis gel, and a legend is intuitive. For machines, however, these composite visualizations are often treated as "black boxes." Standard image segmentation frequently fails because it cannot distinguish between a meaningful sub-figure and "auxiliary" fragments, such as floating legends or axis labels. This difficulty hinders the indexing of visual data and the large-scale analysis of visual communication in science (viziometrics). To address this, we propose a three-step dismantling algorithm that leverages the inherent hierarchical structure of scientific layouts.
Methodology
The dismantling process is divided into three primary stages: recursive splitting, SVM-based merging, and scoring.
1. Recursive Splitting
The algorithm first searches for "fire lanes"—blank horizontal or vertical rows or columns of pixels that act as buffers between sub-figures. By utilizing pixel-value histograms and variance thresholds, the system recursively breaks the image into smaller and smaller blocks based on these whitespace guides.
2. SVM-Based Merging
Recursive splitting often results in fragments (like a title or legend) rather than standalone charts. To resolve this, we trained a Support Vector Machine (SVM) classifier using 15 geometric and whitespace features (e.g., aspect ratio, "blank coverage"). The SVM distinguishes between "standalone" sub-figures and "auxiliary" fragments, allowing the system to merge fragments into their nearest standalone neighbor.
3. Scoring and Selection
Because an image can be split in multiple orders (horizontal vs. vertical first), the system generates various possible decomposition "trees." A custom scoring function that rewards visual balance and repeating patterns is used to select the most likely correct decomposition.
Datasets and Evaluation
The system was trained and evaluated using figures randomly extracted from the PubMed database. The corpus included 1,947 figures for detection training and 213 composite figures (split into 7,541 sub-images) for classifier training. Evaluation was conducted on a set of 261 multi-chart figures containing 1,534 individual visualizations.
Key Results
- Detection Accuracy: The system correctly identified composite vs. single-chart figures with 90.2% accuracy.
- Dismantling Performance: For common cases (eight or fewer sub-figures), the algorithm achieved 80.1% recall and 85.1% precision for perfect extractions.
- Visualization Diversity: The system was 92.3% accurate at identifying the types of visualizations present, even when the extraction was not pixel-perfect.
- Complexity Constraints: Performance dropped to 42.1% recall on "densely packed" figures containing more than nine sub-figures.
Discussion and Future Work
The current algorithm is optimized for grid-aligned layouts. It struggles with irregular "zigzag" boundaries or "homogeneous arrays" (e.g., a grid of 20 nearly identical microscope photos) where the distinction between separate data points and a single observation is ambiguous. Future research will focus on integrating Optical Character Recognition (OCR) to use text location as a layout hint and incorporating color information to better distinguish between adjacent but distinct plots.
Summary updated