A Survey and Approach to Chart Classification

Anurag Dhote, Mohammed Javed, David DoermannLecture Notes in Computer Science 2023 · DOI · OpenAlex

Abstract

Charts represent an essential source of visual information in documents and facilitate a deep understanding and interpretation of information typically conveyed numerically. In the scientific literature, there are many charts, each with its stylistic differences. Recently the document understanding community has begun to address the problem of automatic chart understanding, which begins with chart classification. In this paper, we present a survey of the current state-of-the-art techniques for chart classification and discuss the available datasets and their supported chart types. We broadly classify these contributions as traditional approaches based on ML, CNN, and Transformers. Furthermore, we carry out an extensive comparative performance analysis of CNN-based and transformer-based approaches on the recently published CHARTINFO UB-UNITECH PMC dataset for the CHART-Infographics competition at ICPR 2022. The data set includes 15 different chart categories, including 22,923 training images and 13,260 test images. We have implemented a vision-based transformer model that produces state-of-the-art results in chart classification.

Introduction

"Chart mining"—the process of automatically extracting underlying data and semantics from chart images—is a critical sub-field of document understanding. Before an AI pipeline can extract data, it must first identify the chart type (e.g., bar chart, pie chart, scatter plot). This "chart classification" task is the foundational step for downstream data extraction.

This paper addresses two primary gaps: the lack of a consolidated survey of techniques and datasets from the last decade, and the need for a rigorous comparison between modern Vision Transformer (ViT) architectures and established Convolutional Neural Networks (CNNs) on complex, large-scale datasets.

Literature Survey

The authors categorize historical approaches into three main eras: Traditional ML (utilizing manual features like HOG combined with SVMs), CNN-based Deep Learning (leveraging architectures such as AlexNet, ResNet, and Xception), and the most recent shift toward attention-based Transformer models.

Summary of literature survey on chart classification models and performance.

Datasets

The field has seen a massive scale-up in data availability. The paper catalogs 18 historical datasets, tracing the evolution from early collections of approximately 2,600 images to modern benchmarks exceeding 380,000 images. This growth has been essential for the development of data-hungry deep learning models.

Evolution of chart classification datasets from 2011 to 2022.

For the experimental study, the authors utilize the CHARTINFO UB-PMC dataset (ICPR 2022). This dataset comprises 22,923 training and 13,260 test images across 15 distinct categories, including complex types like Manhattan plots, heatmaps, and radar charts.

Experimental Methodology

To establish modern baselines, the authors fine-tuned several ImageNet-pretrained models for chart classification. The study compared high-performing CNNs (ResNet-152, DenseNet-121, Xception, ConvNeXt) against Vision Transformers (DeIT, Swin Transformer). A key focus was the Swin Transformer, a hierarchical model using shifted windows for self-attention, which is particularly suited for capturing the spatial hierarchies and global context found in charts, such as the relationship between axes and legends.

Key Results and Insights

The experimental results demonstrate a clear advantage for Transformer-based architectures:

  • Swin-Chart Dominance: The best-performing model, SwinL_224 (Swin Large), achieved an F1-score of 0.932, outperforming the ICPR 2022 competition winner (F1: 0.910).
  • CNN Plateau: While modern CNNs like ConvNeXt performed well (F1: 0.896), they lagged significantly behind the Swin architecture, suggesting that standard convolutions may be reaching a performance ceiling for complex chart layouts.
  • Scaling Effects: Large-scale datasets like UB-PMC are essential for training data-hungry Transformer models without severe overfitting, marking a turning point for the field.

Limitations and Future Work

Despite recent progress, several challenges remain. The community lacks a single, unified benchmark covering all possible chart varieties, and niche charts (e.g., Venn diagrams) remain underrepresented compared to common types. Furthermore, future models must handle "noisy" real-world inputs, including background grids, low resolution, and composite charts embedded within complex document layouts.

Summary updated

BibTeX
@inproceedings{dhote2023survey,
  author = {Anurag Dhote and Mohammed Javed and David Doermann},
  title = {{A Survey and Approach to Chart Classification}},
  booktitle = {Lecture Notes in Computer Science},
  year = {2023},
  doi = {10.1007/978-3-031-41498-5\_5},
}

Loading related content...

Figure