Automatic Metadata Extraction Incorporating Visual Features from Scanned Electronic Theses and Dissertations

Muntabir Hasan Choudhury, Himarsha R. Jayanetti, Jian Wu, William A. Ingram, Edward A. Fox

TL;DR

This paper tackles the challenge of automatically extracting metadata from scanned Electronic Theses and Dissertations (ETDs), which often lack proper indexing. They introduce a Conditional Random Field (CRF) model that leverages both text and visual features from document layouts, significantly outperforming previous text-only methods and a heuristic approach.

What Problem Does This Paper Address?

Digital libraries are treasure troves of scholarly knowledge, but their accessibility hinges on good metadata. Electronic Theses and Dissertations (ETDs) are particularly valuable, yet many older ETDs are scanned images rather than born-digital documents. This presents a significant hurdle for existing metadata extraction tools, which are typically designed for text-based documents and struggle with the visual complexities and OCR noise inherent in scanned materials. The authors aim to bridge this gap by developing a robust method for extracting key information like title, author, and degree from these scanned ETD cover pages.

Methods

The core of the paper is a Conditional Random Field (CRF) model. CRFs are a type of statistical model well-suited for sequence labeling tasks, where the goal is to assign a label to each element in a sequence (in this case, each word or token in the document). The authors build upon a standard CRF approach by incorporating novel visual features.

Here's a breakdown:

  • Text-based Features: These are the standard inputs for many NLP tasks, including word identity, Part-of-Speech (POS) tags, and character-level features (like capitalization and length).
  • Visual Features: This is the key innovation. The authors extract visual information from the bounding boxes of text elements as identified by OCR. Specifically, they use normalized x and y coordinates of these bounding boxes (left margin, upper-left corner, bottom-right corner) to capture spatial layout. The intuition is that elements like titles tend to appear in specific regions of a cover page, and this spatial information can be a strong indicator.
  • CRF Model: They employ a BIO (Beginning, Inside, Outside) tagging scheme, where tokens are labeled as the start of a metadata field (B-), inside a metadata field (I-), or outside any field (O-).
  • BiLSTM-CRF Model: As a comparison, they also experiment with a Bidirectional Long Short-Term Memory (BiLSTM) network followed by a CRF layer. BiLSTMs are powerful for capturing sequential context.

Datasets

The authors utilize a newly created dataset for their experiments:

  • 500 ETD Cover Pages: This dataset comprises ETDs from 15 US and 6 non-US universities, published between 1945 and 1990. It includes a mix of STEM and non-STEM majors across doctoral, master's, and bachelor's degrees.
  • Intermediate Datasets: From these 500 ETDs, they derive several intermediate datasets:
  • Original PDF cover pages.
  • TIFF images (preferred for OCR).
  • Raw OCR output (TXT-OCR).
  • Cleaned OCR text (TXT-clean), with spelling corrections and formatting adjustments.
  • Annotated text (TXT-annotated) with human-labeled metadata fields.
  • Ground truth metadata (GT-meta) from library repositories.
  • Revised ground truth metadata (GT-rev) after manual reconciliation of library data with cover page content.

This new dataset is an extension of their previous work, which used a smaller corpus of 100 ETDs.

Key Insights and Ideas

  • Visual Cues Matter: The most significant insight is that visual layout information is crucial for accurately extracting metadata from scanned documents. Simply relying on text alone is insufficient when dealing with the noise and structural variations of scanned pages.
  • CRF with Visual Features is Powerful: Combining text and visual features within a CRF framework provides a robust solution that outperforms simpler methods. The spatial arrangement of text elements offers valuable context that text alone cannot capture.
  • OCR Challenges are Real: The paper highlights several OCR challenges, including scribbles, stamps, overlapping text, and copyright symbols, which can degrade text quality and complicate extraction.
  • Dataset Diversity is Key: The authors note that their heuristic method's performance dropped significantly on the larger, more diverse dataset, underscoring the need for robust, learning-based approaches that can generalize across different institutions and document styles.

Key Results

The proposed CRF model incorporating visual features demonstrated strong performance:

  • Improved Accuracy: The CRF-visual model achieved F1 scores ranging from 81.3% to 96% across seven metadata fields, significantly outperforming both the heuristic baseline and the text-only CRF model.
  • Heuristic Model Performance: The heuristic model, while effective on a smaller, more uniform dataset, saw a considerable drop in accuracy on the larger, more diverse 500-ETD corpus.
  • BiLSTM-CRF Underperformance: The BiLSTM-CRF model yielded poor results, attributed primarily to insufficient training data and potentially the lack of pre-trained language models.

Limitations and Future Work

  • BiLSTM-CRF Limitations: The authors acknowledge that the BiLSTM-CRF model's poor performance was likely due to the small training dataset size and the absence of advanced pre-trained language models.
  • Future Work:
    • Enhance BiLSTM-CRF: They plan to improve the BiLSTM-CRF model by incorporating pre-trained language models like BERT.
  • Post-OCR Error Correction: Integrating a post-OCR error correction step into the pipeline is a key future direction.
  • Direct OCR Input: Running the model directly on raw OCR output (TXT-OCR) instead of cleaned text (TXT-clean) would make the pipeline more end-to-end and potentially more robust to OCR imperfections.

Summary updated

Abstract

Electronic Theses and Dissertations (ETDs) contain domain knowledge that can be used for many digital library tasks, such as analyzing citation networks and predicting research trends. Automatic metadata extraction is important to build scalable digital library search engines. Most existing methods are designed for born-digital documents such as GROBID, CERMINE, and ParsCit, so they often fail to extract metadata from scanned documents such as for ETDs. Traditional sequence tagging methods mainly rely on text-based features. In this paper, we propose a conditional random field (CRF) model that combines text-based and visual features. To verify the robustness of our model, we extended an existing corpus and created a new ground truth corpus consisting of 500 ETD cover pages with human validated metadata. Our experiments show that CRF with visual features outperformed both a heuristic baseline and a CRF model with only text-based features. The proposed model achieved 81.3%-96% F1 measure on seven metadata fields. The data and source code are publicly available on Google Drive11httns://tinvurl.com/y8kxzwrp and a GitHub repository22https://github.com/lamps-lab/ETDMiner/tree/master/etd_crf, respectively.
BibTeX
@inproceedings{choudhury2021automatic,
  author = {Muntabir Hasan Choudhury and Himarsha R. Jayanetti and Jian Wu and William A. Ingram and Edward A. Fox},
  title = {{Automatic Metadata Extraction Incorporating Visual Features from Scanned Electronic Theses and Dissertations}},
  booktitle = {JCDL},
  year = {2021},
  url = {https://openreview.net/forum?id=cRlBdgViSH},
}
Figure