ScanBank: A Benchmark Dataset for Figure Extraction from Scanned Electronic Theses and Dissertations

Sampanna Yashwant Kahu, William A. Ingram, Edward A. Fox, Jian Wu

TL;DR

This paper introduces ScanBank, the first manually annotated dataset for extracting figures and tables from scanned Electronic Theses and Dissertations (ETDs). They demonstrate that existing figure extraction methods, trained on "born-digital" documents, perform poorly on scanned documents and propose a YOLOv5-based model trained on ScanBank that significantly outperforms these baselines.

What Problem Does This Paper Address?

Millions of Electronic Theses and Dissertations (ETDs) are publicly available, forming a vast repository of scholarly knowledge. Figures and tables within these documents are crucial for conveying information concisely. However, a significant portion of older ETDs exist as scanned documents, not born-digital PDFs. Existing state-of-the-art figure extraction methods, which rely on the structural information of born-digital PDFs or are trained on such data, fail to perform well on these scanned documents. This limits the accessibility and utility of valuable information contained within these scanned ETDs. The authors aim to bridge this gap by creating a benchmark dataset and developing robust methods for figure extraction from scanned ETDs.

Methods

The core of the paper involves two main methodological contributions:

  • ScanBank Dataset Creation: The authors manually annotated 10,182 scanned page images from 70 ETDs (published before 1990) to create the ScanBank dataset. This dataset contains labels for 3,375 figures and tables. They also developed specific labeling guidelines to ensure consistency.
  • Data Augmentation for Scanned Document Simulation: To address the performance gap of models trained on born-digital data, they explored data augmentation techniques. These include:
  • Image-based transformations: Random affine rotation, additive Gaussian noise, Gaussian blur, linear contrast, and perspective transform to mimic scanning artifacts.
  • LaTeX-based transformations: Modifying the LaTeX source code to alter font size, font type (typewriter), and line spacing to make born-digital documents visually resemble scanned ones.
  • YOLOv5 Model Training: They trained a YOLOv5 (extra-large version) model on their newly created ScanBank dataset. YOLOv5 is a popular and efficient object detection framework known for its performance.

Datasets

  • ScanBank: The primary contribution is this new, manually annotated benchmark dataset for figure and table extraction from scanned ETDs. It consists of 10,182 scanned page images with 3,375 labeled figures and tables.
  • arXiv Dataset: Used in conjunction with data augmentation techniques to simulate scanned documents for training. This dataset is born-digital.
  • MIT Dataset: The source from which the scanned ETDs for ScanBank were collected.
  • CS-Large and PubMed Datasets: Used for evaluating the performance of the DEEPFIGURES model on born-digital documents.

Key Insights and Ideas

  • Scanned vs. Born-Digital is a Major Hurdle: The stark performance drop of existing methods on scanned ETDs highlights a significant domain shift that current models struggle with. This underscores the need for specialized datasets and training strategies.
  • Data Augmentation Can Bridge the Gap: Simulating scanned document characteristics on born-digital data through image and LaTeX transformations shows promise in improving the performance of existing models.
  • YOLOv5 Excels on Scanned Data: The YOLOv5 architecture, when trained on the ScanBank dataset, demonstrates a substantial leap in performance compared to previous methods, suggesting its suitability for this challenging task.
  • Manual Annotation is Crucial: The creation of ScanBank emphasizes that high-quality, manually annotated data is essential for developing robust models for tasks where automated labeling or existing datasets are insufficient.

Key Results

  • The pre-trained DEEPFIGURES model, when evaluated on the ScanBank dataset, achieved significantly lower F1-scores (around 0.44-0.47) compared to its performance on born-digital datasets (around 0.80-0.85).
  • Data augmentation techniques applied to DEEPFIGURES showed improvements, with models trained using augmented data generally outperforming the original DEEPFIGURES model. Gaussian blur was identified as a particularly helpful transformation.
  • A YOLOv5 model trained on the ScanBank dataset achieved a mean F1-score of 0.860, a substantial improvement over DEEPFIGURES (0.459) on the same dataset.
  • When compared to other commercial and research models (Google AutoML, Azure Custom Vision, Newspaper Navigator), the YOLOv5 model trained on ScanBank also showed superior performance.

Limitations and Future Work

  • Limitations of Data Augmentation: While data augmentation improved performance, the authors note that the leave-one-out ablation study yielded contrasting results, making it difficult to definitively identify the most impactful transformations. They suggest that longer training times might be needed to conclusively assess feature importance.
  • Transfer Learning with DEEPFIGURES: Transfer learning experiments using the DEEPFIGURES model on ScanBank did not improve performance and, in some cases, led to a decrease in the F1-score.
  • Future Work:
    • Doubling the size of the ScanBank dataset.
  • Combining heuristic methods with learning-based approaches.
  • Investigating advanced style transfer techniques (like CycleGANs) to generate scanned ETDs from born-digital ones.
  • Analyzing the differential performance of models on figures versus tables.
  • Developing figure search functionalities for digital libraries.

Summary updated

Abstract

We focus on electronic theses and dissertations (ETDs), aiming to improve access and expand their utility, since more than 6 million are publicly available, and they constitute an important corpus to aid research and education across disciplines. The corpus is growing as new born-digital documents are included, and since millions of older theses and dissertations have been converted to digital form to be disseminated electronically in institutional repositories. In ETDs, as with other scholarly works, figures and tables can communicate a large amount of information in a concise way. Although methods have been proposed for extracting figures and tables from born-digital PDFs, they do not work well with scanned ETDs. Considering this problem, our assessment of state-of-the-art figure extraction systems is that the reason they do not function well on scanned PDFs is that they have only been trained on born-digital documents. To address this limitation, we present ScanBank, a new dataset containing 10 thousand scanned page images, manually labeled by humans as to the presence of the 3.3 thousand figures or tables found therein. We use this dataset to train a deep neural network model based on YOLOv5 to accurately extract figures and tables from scanned ETDs. We pose and answer important research questions aimed at finding better methods for figure extraction from scanned documents. One of those concerns the value for training, of data augmentation techniques applied to born-digital documents which are used to train models better suited for figure extraction from scanned documents. To the best of our knowledge, ScanBank is the first manually annotated dataset for figure and table extraction for scanned ETDs. A YOLOv5-based model, trained on ScanBank, outperforms existing comparable open-source and freely available baseline methods by a considerable margin.
BibTeX
@inproceedings{kahu2021scanbank,
  author = {Sampanna Yashwant Kahu and William A. Ingram and Edward A. Fox and Jian Wu},
  title = {{ScanBank: A Benchmark Dataset for Figure Extraction from Scanned Electronic Theses and Dissertations}},
  booktitle = {JCDL},
  year = {2021},
  url = {https://openreview.net/forum?id=DSKoXFdtdl},
}
Figure