MRN: Multiplexed Routing Network for Incremental Multilingual Text Recognition

Tianlun Zheng, Zhineng Chen, Bingchen Huang, Wei Zhang, Yu-Gang Jiang

Introduction: The Challenge of Incremental Multilingual Text Recognition

Most modern text recognition systems are static, trained once on a fixed set of languages. However, real-world applications often require adding new languages over time without retraining from scratch. This is the domain of Incremental Learning (IL). While IL has seen success in simple image classification, it faces unique hurdles in Incremental Multilingual Text Recognition (IMLTR).

The primary obstacle identified by the authors is rehearsal-imbalance. Standard IL methods often use a small "rehearsal set" of old data to prevent forgetting. In text recognition, languages like Chinese have thousands of characters, and text data naturally follows a long-tail distribution. A small rehearsal set is often unrepresentative, missing many rare characters entirely, which leads to catastrophic forgetting.

Illustration of rehearsal-imbalance where data and class imbalances are aggravated in the rehearsal set.

Methodology: Multiplexed Routing Network (MRN)

To address these challenges, the authors propose the Multiplexed Routing Network (MRN). The framework operates in two distinct stages to ensure both the retention of old knowledge and the acquisition of new scripts.

Stage-I: Individual Language Modeling

When a new language is introduced, a dedicated "expert" recognizer is trained specifically for that language's dataset. This ensures that the model captures the unique features of the new script without interference.

Stage-II: Multi-lingual Modeling and Soft-Voting

Once trained, these language-specific experts are frozen to physically prevent their knowledge from being overwritten. To integrate these experts into a single system, the authors introduce a Domain MLP Router (DM-Router).

The MRN architecture featuring Stage-I individual modeling and Stage-II multi-lingual modeling with a Domain MLP Router.

The DM-Router employs a soft-voting mechanism. Instead of selecting a single expert (hard-voting), it predicts a probability distribution across all known languages. The final output is a weighted combination of predictions from all experts. This allows the model to exploit inter-lingual dependencies—for instance, leveraging shared character structures between Chinese and Japanese to improve overall recognition accuracy.

Experimental Evaluation

The MRN was evaluated on the ICDAR 2017 MLT and ICDAR 2019 MLT benchmarks, which cover a diverse range of scripts including Latin, Arabic, Chinese, Japanese, Korean, and Bangla. The experiments simulated a real-world incremental scenario where languages are added sequentially.

Key Results

MRN demonstrated significant performance gains over state-of-the-art general-purpose incremental learning methods such as LwF, EWC, and DER:

  • Accuracy: MRN achieved average accuracy improvements of 10.3% to 35.8% across different settings.
  • Stability: Unlike other models that saw performance on early tasks plummet as new ones were added, MRN maintained a stable performance curve.
  • Efficiency: The architecture remains scalable, adding minimal computational overhead compared to more complex dynamic expansion models.

Conclusion and Future Directions

MRN represents a significant advancement in applying incremental learning to the complexities of multilingual text. By freezing expert backbones and using a soft-voting router, it effectively mitigates catastrophic forgetting caused by rehearsal-imbalance.

Future research will focus on optimizing the "curriculum" or order in which languages are learned and addressing potential memory overhead as the number of supported languages scales into the hundreds. This work paves the way for more robust, adaptable text recognition systems capable of evolving alongside global linguistic needs.

Summary updated

Abstract

Multilingual text recognition (MLTR) systems typically focus on a fixed set of languages, which makes it difficult to handle newly added languages or adapt to ever-changing data distribution. In this paper, we propose the Incremental MLTR (IMLTR) task in the context of incremental learning (IL), where different languages are introduced in batches. IMLTR is particularly challenging due to rehearsal-imbalance, which refers to the uneven distribution of sample characters in the rehearsal set, used to retain a small amount of old data as past memories. To address this issue, we propose a Multiplexed Routing Network (MRN). MRN trains a recognizer for each language that is currently seen. Subsequently, a language domain predictor is learned based on the rehearsal set to weigh the recognizers. Since the recognizers are derived from the original data, MRN effectively reduces the reliance on older data and better fights against catastrophic forgetting, the core issue in IL. We extensively evaluate MRN on MLT17 and MLT19 datasets. It outperforms existing general-purpose IL methods by large margins, with average accuracy improvements ranging from 10.3% to 35.8% under different settings. Code is available at https://github.com/simplify23/MRN.
BibTeX
@inproceedings{zheng2023mrn,
  author = {Tianlun Zheng and Zhineng Chen and Bingchen Huang and Wei Zhang and Yu-Gang Jiang},
  title = {{MRN: Multiplexed Routing Network for Incremental Multilingual Text Recognition}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2023},
}
Figure