Abstract
For many business applications that require the processing, indexing, and retrieval of professional documents such as legal briefs (in PDF format etc.), it is often essential to classify the pages of any given document into their corresponding types beforehand. Most existing studies in the field of document image classification either focus on single-page documents or treat multiple pages in a document independently. Although in recent years a few techniques have been proposed to exploit the context information from neighboring pages to enhance document page classification, they typically cannot be utilized with large pre-trained language models due to the constraint on input length. In this paper, we present a simple but effective approach that overcomes the above limitation. Specifically, we enhance the input with extra tokens carrying sequential information about previous pages - introducing recurrence - which enables the usage of pre-trained Transformer models like BERT for context-aware page classification. Our experiments conducted on two legal datasets in English and Portuguese respectively show that the proposed approach can significantly improve the performance of document page classification compared to the non-recurrent setup as well as the other context-aware baselines.
Introduction
In the domain of automated document processing, identifying the specific type of a page—such as a "Table of Contents," "Signature Page," or "Evidence Exhibit"—is a critical precursor to downstream tasks like indexing and information extraction. While human readers naturally utilize the sequential flow of a document to categorize pages, modern machine learning models, particularly Large Language Models (LLMs) like BERT, often treat pages as isolated units.
This isolation stems from the inherent architectural constraints of Transformers, which have strict limits on input sequence length. Processing a 100-page legal brief as a single input is computationally prohibitive. Previous attempts to incorporate context have relied on complex architectural additions, such as Conditional Random Fields (CRF) or Long Short-Term Memory (LSTM) layers on top of Transformer outputs. This paper introduces a more streamlined alternative: a recurrent input mechanism that allows BERT to "remember" previous pages without modifying the underlying neural architecture.
Methodology: Recurrent BERT
The proposed approach, termed Recurrent BERT, shifts the complexity from the model architecture to the input representation. The core innovation lies in prepending a special token to the current page's text that represents the classification of the preceding page.
Input Augmentation
For any page $N$ (where $N > 1$), the model takes the predicted label of page $N-1$ and converts it into a dedicated special token (e.g., [type_1], [type_2]). This token is then concatenated to the beginning of the text sequence for page $N$. This allows the Transformer's self-attention mechanism to weigh the previous page's context directly against the current page's content.
Training and Inference
- Teacher Forcing: During the training phase, the model is provided with the ground-truth label of the previous page. This ensures stable training and allows the model to learn the transition probabilities between different page types.
- Sequential Inference: During deployment, the model processes the document chronologically. The prediction for page 1 is used to generate the context token for page 2, and so on.
- Architecture: The method is model-agnostic. In this study, standard BERT was used for English documents and BERTimbau for Portuguese documents, requiring no structural changes to the pre-trained weights.
Experimental Setup
The effectiveness of the recurrent approach was evaluated on two distinct legal benchmarks:
- US Appellate Briefs: A proprietary dataset from Thomson Reuters comprising 781 documents (~22,000 pages). This is a multi-label environment where a single page may belong to multiple categories (e.g., both a "Signature" and a "Certificate of Service").
- SVic+: A public Portuguese dataset containing over 339,000 pages from the Brazilian Supreme Court. This is a single-label dataset featuring six primary document categories.
Results and Discussion
The experimental results demonstrate that incorporating sequential context leads to substantial performance gains over context-oblivious baselines.
Performance Metrics
On the US Appellate Briefs dataset, the recurrent model achieved an F1-score of 97.97%, a significant increase over the 94.71% achieved by standard BERT. The gains were most pronounced in categories that typically appear in specific sequences, such as "Signature" pages (+6.72% F1) and "Other Certificates" (+8.47% F1). On the SVic+ dataset, the model established a new state-of-the-art (SOTA) with a macro-average F1-score of 82.92%, outperforming previous benchmarks by 3.49%.
Context vs. Modality
A notable finding was that this text-only recurrent BERT outperformed LayoutLMv3, a multi-modal model that incorporates visual and layout information. This suggests that for long-form legal documents, the sequential position and the identity of preceding pages are often more predictive of a page's type than its visual appearance.
The "Stickiness" Factor
The model effectively exploits the "stickiness" of certain document sections. For instance, the data shows a 70.62% probability that a page following a "Document Body" page is also a "Document Body" page. By explicitly providing the previous label, the model learns these transition patterns, which are invisible to models processing pages independently.
Conclusion and Future Work
The Recurrent BERT approach offers a simple, "plug-and-play" enhancement for document classification pipelines. By modifying the input string rather than the model architecture, it achieves state-of-the-art results while remaining compatible with existing Transformer frameworks.
However, the approach introduces a sequential bottleneck, as pages must be processed in order, preventing full parallelization of a single document's classification. Additionally, while the model shows robustness, there is a theoretical risk of error propagation if an early page is misclassified. Future research will explore combining this recurrent token method with multi-modal architectures to leverage both visual layout and sequential context simultaneously.
Summary updated