Privacy-Preserving Collaborative Chinese Text Recognition with Federated Learning

Shangchao Su, Haiyang Yu, Bin Li, Xiangyang Xue

Abstract

This paper introduces pFedCR, a personalized federated learning framework specifically designed for Chinese Optical Character Recognition (OCR). Chinese OCR faces significant challenges due to the vast number of character categories and the "long-tail" distribution of data across different organizations. Furthermore, privacy concerns prevent direct data sharing of sensitive documents. pFedCR addresses these issues through a two-stage approach: Global Model Training with server-side virtual data balancing and Local Personalization using attention-based fine-tuning. Experimental results demonstrate a ~20% performance improvement over standalone training and a 6-8% lead over state-of-the-art federated methods, effectively handling "few-shot" characters while maintaining data privacy.

Introduction

Chinese text recognition is a "long-tail" challenge. Unlike Latin scripts with a few dozen characters, Chinese involves thousands of categories. In real-world scenarios—such as medical records or financial invoices—a single organization might only see a small fraction of these characters frequently, leading to poor performance on "few-shot" or rare characters.

Privacy risks and the long-tail distribution of Chinese characters in local datasets.

To improve performance, organizations typically need to pool their data. However, text data is inherently sensitive, often containing addresses, names, and ID numbers. Standard Federated Learning (FL) aims to solve the privacy issue by sharing model weights instead of raw data. However, standard FL often struggles with non-IID data (where different clients have vastly different writing styles) and the massive class imbalance inherent in Chinese text.

Methodology: The pFedCR Framework

The proposed pFedCR (Personalized Federated Chinese Recognition) framework operates in two distinct stages to balance global knowledge with local specialization.

The pFedCR framework overview, illustrating Global Model Training (Stage 1) and Local Personalization (Stage 2).

Stage 1: Global Model Training (GMT)

The framework utilizes a CRNN (Convolutional Recurrent Neural Network) backbone augmented with ECA (Efficient Channel Attention) layers. To combat the long-tail problem, the server uses a public "Text Renderer" to generate a balanced virtual dataset. Before aggregating client weights, the server fine-tunes the global model on this synthetic data to ensure it retains knowledge of rare characters. Additionally, during local training, clients freeze the last embedding layer to prevent local data imbalances from warping the global classification head.

Stage 2: Local Personalization (LPS)

Once a robust global model is established, it is personalized for each client's specific style (e.g., a hospital's specific form layout) through two steps:

  1. Fine-tuning & Averaging: The global model is fine-tuned on local data, and the weights are averaged with the global weights to retain general knowledge.
  2. Attention Fine-tuning: The main CRNN parameters are frozen, and only the ECA attention layers are fine-tuned using a regularization term. This allows the model to adapt to local stylistic features without losing the robust feature extraction capabilities learned during the global stage.

Experimental Evaluation

The study evaluates pFedCR on three industrial-grade datasets: Document (scanned text), Receipt (mobile-captured photos), and HikCSIG (six types of invoices). The researchers simulated real-world institutional silos using both 3-client and 9-client settings.

Key Results and Insights

  • Performance Gains: pFedCR improved local model performance by approximately 20% compared to standalone local training and outperformed other popular personalized FL methods (like FedAvg-ft, FedBN, and Ditto) by 6% to 8%.
  • Solving the Few-Shot Problem: For characters a client had never seen in their local training set, pFedCR achieved accuracies as high as 61.6%, whereas local models scored 0%.
  • Synthetic Data as a Privacy Anchor: By using synthetic data on the server, the model remains balanced without ever seeing private client data. The authors also argue that OCR models are naturally more privacy-resilient than image classifiers due to the sequential and semantic nature of text.
  • Attention for Style: The paper demonstrates that attention layers are the most effective place to inject personalization, allowing the model to adapt to local fonts and backgrounds efficiently.

Conclusion and Future Work

pFedCR provides a strong empirical defense of privacy while significantly improving Chinese OCR performance. Future work may involve implementing formal Differential Privacy (DP) to further harden the framework against potential attacks, as well as adapting the system to open-vocabulary scenarios or extremely rare ancient scripts.

Summary updated

Abstract

In Chinese text recognition, to compensate for the insufficient local data and improve the performance of local few-shot character recognition, it is often necessary for one organization to collect a large amount of data from similar organizations. However, due to the natural presence of private information in text data, such as addresses and phone numbers, different organizations are unwilling to share private data. Therefore, it becomes increasingly important to design a privacy-preserving collaborative training framework for the Chinese text recognition task. In this paper, we introduce personalized federated learning (pFL) into the Chinese text recognition task and propose the pFedCR algorithm, which significantly improves the model performance of each client (organization) without sharing private data. Specifically, pFedCR comprises two stages: multiple rounds of global model training stage and the the local personalization stage. During stage 1, an attention mechanism is incorporated into the CRNN model to adapt to various client data distributions. Leveraging inherent character data characteristics, a balanced dataset is created on the server to mitigate character imbalance. In the personalization phase, the global model is fine-tuned for one epoch to create a local model. Parameter averaging between local and global models combines personalized and global feature extraction capabilities. Finally, we fine-tune only the attention layers to enhance its focus on local personalized features. The experimental results on three real-world industrial scenario datasets show that the pFedCR algorithm can improve the performance of local personalized models by about 20\% while also improving their generalization performance on other client data domains. Compared to other state-of-the-art personalized federated learning methods, pFedCR improves performance by 6\% $\sim$ 8\%.
BibTeX
@misc{su2023privacy,
  author = {Shangchao Su and Haiyang Yu and Bin Li and Xiangyang Xue},
  title = {{Privacy-Preserving Collaborative Chinese Text Recognition with Federated Learning}},
  year = {2023},
  eprint = {2305.05602},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2305.05602},
}
Figure