Abstract
Electronic theses and dissertations (ETDs) have been proposed, advocated, and generated for more than 25 years. Although ETDs are hosted by commercial or institutional digital library repositories, they are still an understudied type of scholarly big data, partially because they are usually longer than conference and journal papers. Segmenting ETDs will allow researchers to study sectional content. Readers can navigate to particular pages of interest, to discover and explore the content buried in these long documents. Most existing frameworks on document page classification are designed for classifying general documents, and perform poorly on ETDs. In this paper, we propose ETDPC. Its backbone is a two-stream multimodal model with a cross-attention network to classify ETD pages into 13 categories. To overcome the challenge of imbalanced labeled samples, we augmented data for minority categories and employed a hierarchical classifier. ETDPC outperforms the state-of-the-art models in all categories, achieving an F1 of 0.84 -- 0.96 for 9 out of 13 categories. We also demonstrated its data efficiency.
Introduction
Electronic Theses and Dissertations (ETDs) represent a unique and vast repository of scholarly knowledge. Unlike standard 10-page conference papers, ETDs are often book-length documents, spanning 100 to 400 pages, and containing specialized sections such as dedications, lists of tables, and complex appendices. Despite their value, they remain an understudied form of scholarly big data due to the difficulty of navigating and segmenting such long-form content.
Existing state-of-the-art (SOTA) document classifiers, such as LayoutLMv2 or DocFormer, are typically pre-trained on general business documents like emails, forms, and receipts. When applied to the academic structure of ETDs, these models struggle significantly, often achieving accuracy as low as 9%. The challenges are rooted in structural complexity—including varied resolutions and scanning noise—and extreme data imbalance, where "Chapter" pages vastly outnumber minority classes like "Dedications" or "Curriculum Vitae."
The ETDPC Framework
To address these challenges, we propose the ETD Page Classifier (ETDPC). ETDPC utilizes a "top-down" multimodal approach that integrates both visual and textual signals through a two-stream architecture.
Architecture and Fusion
The model processes page images through a ResNet-50v2 vision encoder and page text through a specialized BERT with Talking-Heads Attention text encoder. Rather than simply concatenating these features, ETDPC employs a cross-attention network. This mechanism allows the model to align the layout with the content by "focusing" on specific image regions that correspond to specific words in the text.
Hierarchical Classification
Given that the majority of pages in an ETD are chapters, a standard flat classifier often ignores minority classes. ETDPC uses a hierarchical strategy: it first distinguishes between "Chapter" and "Non-Chapter" pages, and then sub-classifies the non-chapter pages into 12 specific categories, such as Abstract, Table of Contents, and CV.
Data Augmentation and the ETD500 Dataset
A critical contribution of this work is the ETD500 dataset, which contains 92,371 manually annotated pages from 500 scanned ETDs published between 1945 and 1990. To handle the class imbalance, the researchers developed an innovative synthetic data generation pipeline.
Using the PEGASUS transformer model, they paraphrased existing text to create new linguistic variations of minority page types. This text was then converted into images using various fonts and subjected to "ScanBank" transformations—including Gaussian noise, salt-and-pepper noise, and blur—to create realistic "pseudo-ETD" pages. This process added 5,984 samples to bolster the training of minority classes, demonstrating that NLP paraphrasing can effectively enhance computer vision tasks.
Key Results and Discussion
ETDPC achieved a breakthrough in performance, reaching 85% accuracy and a 0.83 macro F1 score. This is a massive improvement over generalist models like LayoutLMv2-base (9% accuracy) and DocFormer (28% accuracy).
Performance Insights
The ablation studies highlighted the importance of the specialized components. Using "Talking-Heads Attention" in the BERT encoder provided a significant boost, improving F1 scores by up to 0.43 in certain categories compared to standard BERT-large. Furthermore, the model proved highly efficient, processing a single page in just 0.06 seconds, making it scalable for university libraries holding millions of documents.
Visual vs. Textual Signals
The research found that while text is a strong indicator for "Chapters," visual layout is often the primary signal for identifying pages like "List of Figures" or "Title Pages," where spatial arrangement is more diagnostic than the specific words used. However, the model still faces challenges in distinguishing between "List of Figures" and "List of Tables" due to their near-identical visual and linguistic profiles.
Conclusion and Future Work
ETDPC establishes a new benchmark for academic document classification. By specializing the architecture for the long-form structure of ETDs and employing clever data augmentation, the framework overcomes the limitations of general-purpose document AI. Future work will focus on adding heuristic-based pattern matchers to resolve visual similarity confusion and adopting multilingual models to support international dissertation repositories. This tool provides a low-cost, high-efficiency path for university libraries to index and unlock the content within their digital commons.
Summary updated