Rvafm: Re-Parameterizing Vertical Attention Fusion Module for Handwritten Paragraph Text Recognition

Jinhui Zheng, Zhiquan Liu, Yain‐Whar Si, Jianqing Li, Xinyuan Zhang, Xiaofan Li, Haozhi Huang, Xueyuan Gong

TL;DR

This paper introduces RVAFM, a novel module for handwritten paragraph text recognition (HPTR) that decouples training and inference stages. By employing a multi-branch structure during training for enhanced learning and a single-branch structure during inference for speed, RVAFM achieves state-of-the-art results on the IAM dataset while maintaining efficient inference.

What Problem Does This Paper Address?

Handwritten Paragraph Text Recognition (HPTR) is a challenging task that involves converting an entire paragraph of handwritten text into a sequence of characters. Existing advanced models like the Vertical Attention Network (VAN) use Vertical Attention Modules (VAMs) to implicitly segment paragraphs into text lines. However, VAMs, being single-branch modules, have limited expressive power compared to multi-branch architectures. Simply converting VAM to a multi-branch structure during inference would significantly slow down processing. This paper addresses the dilemma of achieving both high accuracy and fast inference in HPTR by proposing a method that optimizes the module's structure for each stage.

Methods

The core innovation is the Re-parameterizing Vertical Attention Fusion Module (RVAFM), which leverages structural re-parameterization techniques.

  • Dual-Parameter Layers: RVAFM introduces "dual-parameter layers" which are essentially two parallel layers with identical configurations. During training, these layers operate in parallel, and their outputs are summed. This creates a multi-branch structure.
  • Training Stage: In the training phase, RVAFM utilizes a multi-branch structure composed of these dual-parameter layers. This allows for more complex feature learning and improved model expressiveness. The paper mentions RVAFM applies a total of five dual-parameter layers, resulting in a 2^5 = 32-branch structure.
  • Inference Stage: For inference, RVAFM employs a Re-parameterization Fusion (RF) method. This method fuses the parameters of the dual-parameter layers into a single-parameter layer. The fusion is based on the distributive property of convolution and matrix multiplication, ensuring no loss of information or accuracy. This results in a lightweight, single-branch structure for faster processing.
  • Decoupled Structure: The key idea is to decouple the module's structure between training and inference. The multi-branch structure during training enhances learning capability, while the fused single-branch structure during inference optimizes speed.

Datasets

  • IAM Dataset: The experiments are conducted on the IAM dataset, which contains handwritten copies of text paragraphs from the LOB corpus. This dataset provides segmentation and transcriptions at various levels (page, paragraph, line, word). The paper uses an unofficial split for training (747 images), validation (116 images), and testing (336 images).

Key Insights and Ideas

  • Training vs. Inference Optimization: The paper highlights that optimizing a model for training (e.g., complex multi-branch structures for learning) is often at odds with optimizing for inference (e.g., simple single-branch structures for speed). Structural re-parameterization offers a way to bridge this gap.
  • Robustness through Redundancy: The dual-parameter layer design, with its two parallel branches, offers increased robustness. If one branch encounters learning errors, the other can compensate, reducing the impact on earlier layers and enhancing the overall learning process.
  • Breadth and Depth in Learning: The dual-parameter layers facilitate both broadening (establishing distinct functions) and deepening (strengthening existing functions) of a layer's capabilities during training, leading to improved overall model accuracy.
  • Lossless Fusion: The Re-parameterization Fusion (RF) method is crucial, as it ensures that the complex multi-branch structure learned during training can be converted into an efficient single-branch structure for inference without any loss in accuracy.

Key Results

  • State-of-the-Art Performance: The proposed RVAN (VAN with RVAFM) achieves state-of-the-art results on the IAM paragraph-level test set.
  • CER: 4.44% (compared to the best previous result of 4.4% from Bluche [13]).
  • WER: 14.37% (surpassing most methods, with Bluche [13] achieving 10.9%).
  • Improved Accuracy over VAN: RVAN outperforms the original VAN in both CER (4.40% vs. 4.45%) and WER (14.37% vs. 14.55%) when comparing RVAN-T (RVAN before fusion) to VAN.
  • Faster Inference: RVAN-I (RVAN after fusion) achieves a slightly faster inference speed (18.3 ms) compared to VAN (19.3 ms) and RVAN-T (19.6 ms), while maintaining accuracy.
  • Parameter Efficiency: The fused RVAN-I model has fewer parameters (2.6 MB) compared to VAN (2.7 MB) and RVAN-T (2.7 MB).

Limitations and Future Work

  • Hyper-parameter Sensitivity: The paper explores hyper-parameter tuning for $C_u$ (channel number for attention) and NSL (number of sub-layers). While RVAN-Cu256 showed good performance, RVAN-Cu512, despite a larger $C_u$, did not yield the best results, suggesting potential challenges with very large channel numbers. Similarly, increasing NSL beyond 2 did not significantly improve performance.
  • Future Research: The authors suggest designing a training strategy that dynamically adjusts dual-parameter layers based on the states of different modules to further improve model training effectiveness.

Summary updated

Abstract

Handwritten Paragraph Text Recognition (HPTR) is a challenging task in Computer Vision, requiring the transformation of a paragraph text image, rich in handwritten text, into text encoding sequences. One of the most advanced models for this task is Vertical Attention Network (VAN), which utilizes a Vertical Attention Module (VAM) to implicitly segment paragraph text images into text lines, thereby reducing the difficulty of the recognition task. However, from a network structure perspective, VAM is a single-branch module, which is less effective in learning compared to multi-branch modules. In this paper, we propose a new module, named Re-parameterizing Vertical Attention Fusion Module (RVAFM), which incorporates structural re-parameterization techniques. RVAFM decouples the structure of the module during training and inference stages. During training, it uses a multi-branch structure for more effective learning, and during inference, it uses a single-branch structure for faster processing. The features learned by the multi-branch structure are fused into the single-branch structure through a special fusion method named Re-parameterization Fusion (RF) without any loss of information. As a result, we achieve a Character Error Rate (CER) of 4.44% and a Word Error Rate (WER) of 14.37% on the IAM paragraph-level test set. Additionally, the inference speed is slightly faster than VAN.
BibTeX
@article{zheng2026rvafm,
  author = {Jinhui Zheng and Zhiquan Liu and Yain‐Whar Si and Jianqing Li and Xinyuan Zhang and Xiaofan Li and Haozhi Huang and Xueyuan Gong},
  title = {{Rvafm: Re-Parameterizing Vertical Attention Fusion Module for Handwritten Paragraph Text Recognition}},
  journal = {Information Fusion},
  year = {2026},
  doi = {10.1016/j.inffus.2025.103491},
}
Figure