A Systematic Evaluation of Object Detection Networks for Scientific Plots

Pritha Ganguly, Nitesh S. Methani, Mitesh M. Khapra, Pratyush Kumar

Introduction

While object detection has made significant strides in identifying objects like cats and cars in natural images, "reading" a scientific plot presents a unique set of challenges for standard AI models. In their AAAI-21 paper, researchers from IIT Madras demonstrate that while models like YOLO or Faster R-CNN might appear successful on a graph at first glance, they often fail in the critical area of localization precision.

The Precision Gap in Plot Detection

In natural image detection, an Intersection over Union (IoU) of 0.5 is typically considered acceptable. If a bounding box for a dog is slightly off, the classification remains correct. However, in the context of scientific plots, a "slightly off" bounding box is catastrophic. If a model detects a bar in a chart but misses the top edge by just a few pixels, the subsequent numerical data extraction will be inaccurate.

Comparison of IoU requirements between natural images (PASCAL VOC) and scientific plots (PlotQA).

The authors found that while existing models achieve approximately 80% mean Average Precision (mAP) at the standard 0.5 IoU, their performance drops significantly—to as low as 35%—when evaluated at a strict 0.9 IoU threshold. Scientific plots also introduce structural challenges such as extreme aspect ratios for bars and the need to link legend previews with their corresponding text.

PlotNet: A Specialized Architecture

To address these issues, the authors proposed PlotNet, a "bottom-up" architecture designed for the geometric nature of computer-generated plots. Unlike general-purpose models, PlotNet incorporates several domain-specific innovations:

  • Laplacian Region Proposals: Instead of a deep learning-based Region Proposal Network (RPN), PlotNet utilizes a classic Laplacian edge detector. This traditional computer vision method is more precise for clean-edged plots and significantly faster, reducing proposal time from 6740ms to just 34ms.
  • AN-ROI Layer: The "Accumulating Neighbouring ROI" layer allows the network to incorporate context from the immediate surroundings of a proposal. This helps the model identify elements like tick marks or labels that define a box's identity.
  • Linking Component: This module predicts whether adjacent proposals should be merged, effectively handling long text strings like titles that might otherwise be fragmented.
  • Focal IoU (FIOU) Loss: A custom loss function that penalizes localization errors more heavily as overlap increases, forcing the model to align bounding boxes perfectly with visual edges.

Datasets and Evaluation

The research primarily utilizes PlotQA, a massive dataset of over 220,000 scientific plots (bar, line, and scatter) derived from real-world sources like World Bank reports. PlotQA is notably more challenging than synthetic datasets like FigureQA or DVQA because it uses real-world variable names and values.

Key Results

PlotNet demonstrates superior performance across several metrics:

  • Accuracy: It achieved 93.44% mAP @ 0.9 IoU, outperforming the best hybrid models (Faster R-CNN with FPN and ROIAlign) by over 16 points.
  • Efficiency: PlotNet is 16x faster than its closest competitor with 3x lower FLOPs, making it suitable for real-time applications.
  • Data Extraction: In "plot-to-table" tasks, PlotNet achieved an F1 score of 0.77, a 35% relative improvement over previous state-of-the-art methods.

Conclusion

PlotNet highlights that in structured, non-natural domains, traditional computer vision techniques can complement deep learning to achieve the high precision required for data extraction. While the model excels at computer-generated plots, future work will likely focus on robustness against noisy or scanned documents and advancing automated reasoning to answer complex questions about the trends identified within these plots.

Summary updated

Abstract

Are existing object detection methods adequate for detecting text and visual elements in scientific plots which are arguably different than the objects found in natural images? To answer this question, we train and compare the accuracy of various SOTA object detection networks on the PlotQA dataset. At the standard IOU setting of 0.5, most networks perform well with mAP scores greater than 80% in detecting the relatively simple objects in plots. However, the performance drops drastically when evaluated at a stricter IOU of 0.9 with the best model giving a mAP of 35.70%. Note that such a stricter evaluation is essential when dealing with scientific plots where even minor localisation errors can lead to large errors in downstream numerical inferences. Given this poor performance, we propose minor modifications to existing models by combining ideas from different object detection networks. While this significantly improves the performance, there are still 2 main issues: (i) performance on text objects which are essential for reasoning is very poor, and (ii) inference time is unacceptably large considering the simplicity of plots. To solve this open problem, we make a series of contributions: (a) an efficient region proposal method based on Laplacian edge detectors, (b) a feature representation of region proposals that includes neighbouring information, (c) a linking component to join multiple region proposals for detecting longer textual objects, and (d) a custom loss function that combines a smooth L1-loss with an IOU-based loss. Combining these ideas, our final model is very accurate at extreme IOU values achieving a mAP of 93.44%@0.9 IOU. Simultaneously, our model is very efficient with an inference time 16x lesser than the current models, including one-stage detectors. With these contributions, we enable further exploration on the automated reasoning of plots.
BibTeX
@inproceedings{ganguly2021systematic,
  author = {Pritha Ganguly and Nitesh S. Methani and Mitesh M. Khapra and Pratyush Kumar},
  title = {{A Systematic Evaluation of Object Detection Networks for Scientific Plots}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2021},
  doi = {10.1609/aaai.v35i2.16227},
}
Figure