Information Block Detection in Infographic Based on Spatial Proximity and Structural Similarity (Student Abstract)

Jie Lin, Xin Wu, Jianwei Lu, Yi CaiAAAI 2021 · DOI

Abstract

Infographics are complex visual summaries where the spatial arrangement of elements like icons, titles, and text conveys logical meaning. Automatically grouping these elements into coherent "information blocks" is a fundamental step for machine understanding of infographics. This paper proposes the SS-FE (Spatial and Structural Feature Extraction) model, which enhances traditional proximity-based grouping by incorporating structural cues such as background colors and boundaries. Experimental results on a large-scale infographic dataset demonstrate that SS-FE achieves a state-of-the-art block accuracy of 0.5614, outperforming distance-only baselines.

Introduction

Infographics present a unique challenge for computer vision because their layout is often as informative as their content. To interpret an infographic, a system must first identify which elements belong together in an "information block." Traditionally, research has leaned on the Gestalt principle of spatial proximity—the assumption that elements close to each other are logically related. However, professional designers frequently use structural dividers, such as sharp color transitions or thin lines, to separate elements that are physically adjacent but conceptually distinct. Models relying solely on distance often fail in these scenarios, leading to incorrect merges. The SS-FE model aims to bridge this gap by explicitly modeling structural similarity alongside spatial proximity.

Methods: The SS-FE Model

The authors frame block detection as a pair-wise classification problem: given two elements, do they belong to the same information block? The proposed SS-FE model utilizes a dual-branch architecture to answer this question.

The SS-FE model architecture featuring spatial and structural feature extraction branches.
  1. Spatial Feature Extractor: This branch uses a ResNet backbone to analyze a cropped image patch containing the pair of elements. It captures relative positional features, identifying relationships such as overlap or adjacency.
  2. Structural Feature Extractor: This branch operates on an "annotation map" where element types and boundaries are highlighted. It employs a Feature Pyramid Network (FPN) to extract multi-scale structural features, allowing the model to detect both fine-grained dividers (lines) and global layout structures (color regions).
  3. Adaptive Fusion: Features from both branches are fused using adaptive max-pooling and a Multi-Layer Perceptron (MLP) enhanced with an attention mechanism.

The resulting pair-wise predictions are then processed through a seeding and grouping strategy to form final, multi-element blocks.

Dataset and Evaluation

The model was trained and tested on a benchmark derived from an existing infographic dataset (Lu et al. 2020).

  • Scale: 1,417 infographics.
  • Annotations: 149,511 element pairs with ground-truth labels.
  • Element Types: Icons, titles, text, and marks.

Performance is measured using a Jaccard-based "block accuracy" metric (block_acc).

Key Results

The SS-FE model demonstrates a significant performance leap over previous methods. As shown in the results table, the full SS-FE model achieves the highest accuracy and F1 scores.

Performance comparison of the SS-FE model against the baseline and an ablation version.

The ablation study ("SS-FE \wo st") highlights the critical role of structural features. Removing the structural branch causes the block_acc to drop from 0.5614 to 0.5062. This confirms that while spatial proximity is a strong indicator, structural cues act as essential "visual walls" that refine the logical grouping of elements.

Conclusion

The SS-FE model successfully integrates spatial and structural information to improve block detection in infographics. By treating background colors and boundaries as explicit signals, the model overcomes the limitations of purely distance-based approaches. Future research may involve integrating semantic text analysis (NLP) to further improve grouping accuracy and exploring the model's robustness against highly textured or photographic backgrounds.

Summary updated

BibTeX
@inproceedings{lin2021information,
  author = {Jie Lin and Xin Wu and Jianwei Lu and Yi Cai},
  title = {{Information Block Detection in Infographic Based on Spatial Proximity and Structural Similarity (Student Abstract)}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2021},
  doi = {10.1609/aaai.v35i18.17911},
}

Loading related content...

Figure