TL;DR
This paper introduces ScanBank, the first manually annotated dataset for extracting figures and tables from scanned Electronic Theses and Dissertations (ETDs). They demonstrate that existing figure extraction methods, trained on "born-digital" documents, perform poorly on scanned documents and propose a YOLOv5-based model trained on ScanBank that significantly outperforms these baselines.
What Problem Does This Paper Address?
Millions of Electronic Theses and Dissertations (ETDs) are publicly available, forming a vast repository of scholarly knowledge. Figures and tables within these documents are crucial for conveying information concisely. However, a significant portion of older ETDs exist as scanned documents, not born-digital PDFs. Existing state-of-the-art figure extraction methods, which rely on the structural information of born-digital PDFs or are trained on such data, fail to perform well on these scanned documents. This limits the accessibility and utility of valuable information contained within these scanned ETDs. The authors aim to bridge this gap by creating a benchmark dataset and developing robust methods for figure extraction from scanned ETDs.
Methods
The core of the paper involves two main methodological contributions:
- ScanBank Dataset Creation: The authors manually annotated 10,182 scanned page images from 70 ETDs (published before 1990) to create the ScanBank dataset. This dataset contains labels for 3,375 figures and tables. They also developed specific labeling guidelines to ensure consistency.
- Data Augmentation for Scanned Document Simulation: To address the performance gap of models trained on born-digital data, they explored data augmentation techniques. These include:
- Image-based transformations: Random affine rotation, additive Gaussian noise, Gaussian blur, linear contrast, and perspective transform to mimic scanning artifacts.
- LaTeX-based transformations: Modifying the LaTeX source code to alter font size, font type (typewriter), and line spacing to make born-digital documents visually resemble scanned ones.
- YOLOv5 Model Training: They trained a YOLOv5 (extra-large version) model on their newly created ScanBank dataset. YOLOv5 is a popular and efficient object detection framework known for its performance.
Datasets
- ScanBank: The primary contribution is this new, manually annotated benchmark dataset for figure and table extraction from scanned ETDs. It consists of 10,182 scanned page images with 3,375 labeled figures and tables.
- arXiv Dataset: Used in conjunction with data augmentation techniques to simulate scanned documents for training. This dataset is born-digital.
- MIT Dataset: The source from which the scanned ETDs for ScanBank were collected.
- CS-Large and PubMed Datasets: Used for evaluating the performance of the DEEPFIGURES model on born-digital documents.
Key Insights and Ideas
- Scanned vs. Born-Digital is a Major Hurdle: The stark performance drop of existing methods on scanned ETDs highlights a significant domain shift that current models struggle with. This underscores the need for specialized datasets and training strategies.
- Data Augmentation Can Bridge the Gap: Simulating scanned document characteristics on born-digital data through image and LaTeX transformations shows promise in improving the performance of existing models.
- YOLOv5 Excels on Scanned Data: The YOLOv5 architecture, when trained on the ScanBank dataset, demonstrates a substantial leap in performance compared to previous methods, suggesting its suitability for this challenging task.
- Manual Annotation is Crucial: The creation of ScanBank emphasizes that high-quality, manually annotated data is essential for developing robust models for tasks where automated labeling or existing datasets are insufficient.
Key Results
- The pre-trained DEEPFIGURES model, when evaluated on the ScanBank dataset, achieved significantly lower F1-scores (around 0.44-0.47) compared to its performance on born-digital datasets (around 0.80-0.85).
- Data augmentation techniques applied to DEEPFIGURES showed improvements, with models trained using augmented data generally outperforming the original DEEPFIGURES model. Gaussian blur was identified as a particularly helpful transformation.
- A YOLOv5 model trained on the ScanBank dataset achieved a mean F1-score of 0.860, a substantial improvement over DEEPFIGURES (0.459) on the same dataset.
- When compared to other commercial and research models (Google AutoML, Azure Custom Vision, Newspaper Navigator), the YOLOv5 model trained on ScanBank also showed superior performance.
Limitations and Future Work
- Limitations of Data Augmentation: While data augmentation improved performance, the authors note that the leave-one-out ablation study yielded contrasting results, making it difficult to definitively identify the most impactful transformations. They suggest that longer training times might be needed to conclusively assess feature importance.
- Transfer Learning with DEEPFIGURES: Transfer learning experiments using the DEEPFIGURES model on ScanBank did not improve performance and, in some cases, led to a decrease in the F1-score.
- Future Work:
- Doubling the size of the ScanBank dataset.
- Combining heuristic methods with learning-based approaches.
- Investigating advanced style transfer techniques (like CycleGANs) to generate scanned ETDs from born-digital ones.
- Analyzing the differential performance of models on figures versus tables.
- Developing figure search functionalities for digital libraries.
Summary updated