CSSL-MHTR: Continual Self-Supervised Learning for Scalable Multi-script Handwritten Text Recognition

Yuyang Liu 刘宇阳arXiv 2023 · DOI · arXiv

TL;DR

This paper introduces CSSL-MHTR, a novel framework for continual self-supervised learning (SSL) in handwritten text recognition (HTR). It addresses the challenge of catastrophic forgetting in HTR models as they learn new scripts and languages sequentially, achieving state-of-the-art performance while being computationally efficient.

What Problem Does This Paper Address?

Handwritten Text Recognition (HTR) systems are crucial for digitizing historical documents and various forms of text. Traditional HTR models often rely on large amounts of labeled data, which is expensive and time-consuming to acquire. Self-supervised learning (SSL) has emerged as a powerful alternative, enabling models to learn from unlabeled data. However, a significant challenge in real-world scenarios is the continuous emergence of new data and tasks (e.g., new scripts or languages). Existing SSL methods typically assume all data is available at once. When new data arrives sequentially, models trained on it tend to forget previously learned information, a phenomenon known as "catastrophic forgetting." Retraining models from scratch on all accumulated data is impractical due to computational costs and potential privacy concerns. This paper tackles the problem of enabling HTR models to learn new scripts and languages sequentially without forgetting previous ones, using an SSL approach.

Methods

The proposed CSSL-MHTR framework operates in two main phases: pre-training and fine-tuning.

Pre-training: The core of the pre-training is a Masked Autoencoder (MAE)-based SSL approach. The model learns to reconstruct masked patches of handwritten text images. This is a departure from traditional supervised learning and allows for learning rich representations from unlabeled data.

To address catastrophic forgetting, two key techniques are employed:

  • Task-Specific Adapters: For each new script/language (task), small, trainable "adapter" modules are added to the existing encoder-decoder architecture. Crucially, all previously trained components of the model are frozen. This ensures that prior knowledge is preserved while only the adapters learn the new task. These adapters are inserted after the Multi-Head Attention (MHA) and feed-forward layers.
  • Memory Replay with Knowledge Distillation: To further improve performance on previous tasks and mitigate forgetting, a memory buffer stores a subset of patches from previously learned tasks. When training on a new task, this stored data is replayed. A knowledge distillation loss is used to ensure the current model's output on these replayed samples matches the output of the previous task's model. A significant advantage here is that only patches, not full images, need to be stored, reducing memory requirements.

Fine-tuning: After the continual self-supervised pre-training, the learned encoder can be fine-tuned on specific downstream HTR tasks for individual scripts or languages. The pre-trained encoder is designed to be generalizable, allowing it to serve as an optimal starting point for various HTR tasks.

Datasets

The experiments were conducted using the following datasets:

  • IAM Dataset: For English handwritten text.
  • LAM Dataset: For Italian handwritten text, specifically ancient manuscripts.
  • HKR Dataset: For Russian handwritten text.

The paper notes that for the first pre-trained model, an additional 40k synthetic lines of English text were used.

Key Insights and Ideas

  • SSL for Continual Learning in HTR is Novel: The paper claims to be the first to apply continual self-supervised learning to handwritten text recognition, opening a new avenue for research in this domain.
  • Adapters are Key to Preserving Knowledge: Freezing the main model and only training task-specific adapters is a highly effective strategy for preventing catastrophic forgetting in SSL.
  • Patch-Based Memory Replay is Efficient: Storing only patches from previous tasks, rather than full images, significantly reduces memory footprint, making continual learning more practical, especially in privacy-sensitive scenarios.
  • Richer Representations Improve Performance: As the model learns more scripts/languages, the learned representations become richer and more robust, leading to improved performance even on previously learned tasks. This is observed as a decrease in CER for English and Italian after learning Russian.
  • Task-Agnostic Inference: The adapter-based approach, combined with memory replay, allows for task-agnostic inference, meaning the model doesn't need explicit task identification at inference time.

Key Results

  • State-of-the-Art Performance: CSSL-MHTR achieved state-of-the-art results on English, Italian, and Russian HTR tasks after continual pre-training, outperforming several baseline continual learning methods.
  • Reduced Character Error Rate (CER): For example, after learning the third task (Russian), the CER for English was 4.9% and for Italian was 5.1%, demonstrating strong performance and minimal forgetting.
  • Computational Efficiency: The adapter-based approach significantly reduces the number of trainable parameters, leading to more efficient training. The paper highlights that CSSL-MHTR requires only 10.2M trainable parameters for the entire continual learning process, compared to much larger numbers for other methods.
  • Robustness and Generalization: The model showed robust performance across different scripts and languages, demonstrating good generalization capabilities. Qualitative results showed that the model could reconstruct masked images from previous tasks with comparable quality to the original task model.

Limitations and Future Work

  • Limitations:
    • The paper acknowledges that while the adapter approach prevents forgetting, the adaptation of the current adapter to previous tasks might be insufficient, leading to some performance degradation compared to a dedicated monolingual model.
  • The memory replay buffer size needs to be managed, and the paper proposes an equal sharing strategy for subsequent tasks.
  • Future Work:
    • Extend the approach to recognize full pages instead of just segmented lines.
  • Incorporate other document analysis tasks, such as layout analysis, name entity recognition, and information extraction, into the continual learning framework.

Summary updated

Abstract

Self-supervised learning has recently emerged as a strong alternative in document analysis. These approaches are now capable of learning high-quality image representations and overcoming the limitations of supervised methods, which require a large amount of labeled data. However, these methods are unable to capture new knowledge in an incremental fashion, where data is presented to the model sequentially, which is closer to the realistic scenario. In this paper, we explore the potential of continual self-supervised learning to alleviate the catastrophic forgetting problem in handwritten text recognition, as an example of sequence recognition. Our method consists in adding intermediate layers called adapters for each task, and efficiently distilling knowledge from the previous model while learning the current task. Our proposed framework is efficient in both computation and memory complexity. To demonstrate its effectiveness, we evaluate our method by transferring the learned model to diverse text recognition downstream tasks, including Latin and non-Latin scripts. As far as we know, this is the first application of continual self-supervised learning for handwritten text recognition. We attain state-of-the-art performance on English, Italian and Russian scripts, whilst adding only a few parameters per task. The code and trained models will be publicly available.
BibTeX
@misc{2023cssl,
  author = {Yuyang Liu 刘宇阳},
  title = {{CSSL-MHTR: Continual Self-Supervised Learning for Scalable Multi-script Handwritten Text Recognition}},
  year = {2023},
  eprint = {2303.09347},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2303.09347},
}
Figure