Charts are the primary medium for communicating data in financial reports, news articles, and scientific papers. However, the underlying numerical data is often "trapped" in static image formats like PNG or JPEG, making it inaccessible for automated analysis or screen readers. ChartOCR is a hybrid framework designed to solve this problem by extracting raw data from chart images using a combination of deep-learning-based keypoint detection and traditional rule-based logic.
The Challenge of Chart Data Extraction
Existing solutions for reverse-engineering charts typically fall into two categories: brittle rule-based methods that fail on diverse styles, and "black-box" deep learning models that lack intermediate control. ChartOCR bridges this gap by treating chart elements as sets of semantically meaningful points rather than simple bounding boxes.
Methodology: A Three-Stage Pipeline
The researchers proposed a three-stage "Deep Hybrid" pipeline to convert pixels back into structured tables:
- Common Information Extraction: Instead of standard object detection, the model uses Keypoint Detection. Utilizing a modified CornerNet with an Hourglass backbone, it identifies critical points: corners for bars, pivot points for lines, and center/arc intersections for pies. This stage also classifies the chart type.
- Data Range Extraction: To map pixel coordinates to actual values, the system uses the Microsoft OCR API to read axis labels and applies a geometric algorithm to determine the numerical scale of the plot area.
- Type-Specific Object Detection: Once keypoints are identified, the system uses specific rules to group them into data series. For example, it uses Associative Embedding to link points belonging to the same line and a Sector Combining algorithm to handle standard or "exploded" pie charts.
Key Insights and Innovations
- Keypoints as a Universal Language: By focusing on coordinates rather than shapes, the framework creates a unified vision front-end that generalizes across bar, line, and pie charts.
- Hybridization for Logic: While deep learning handles the complex vision task of finding points, rule-based logic ensures the data follows chart conventions (e.g., pie slices summing to 100%).
- Real-World Data: The model was trained on ExcelChart400K, a new dataset of nearly 387,000 real-world chart images crawled from the web, ensuring it can handle the variety and complexity of professional designs.
Results and Performance
ChartOCR significantly outperformed previous state-of-the-art methods. On the ExcelChart400K dataset, it achieved accuracy scores between 0.92 and 0.96. It showed a 14.5% improvement in bar chart extraction and a 50% improvement in line chart extraction compared to previous baselines. Furthermore, the system is highly efficient, processing images in 0.2 to 0.5 seconds.
Limitations and Future Directions
Despite its success, the framework faces challenges with highly entangled line charts where overlapping series can lead to point assignment errors. It also remains dependent on the accuracy of the external OCR engine for reading axis labels. Future work will focus on expanding the framework to handle more complex visualizations, such as scatter plots and radar charts.
Summary updated