Research on Multi-Path Feature Fusion Manchu Recognition Based on Swin Transformer

Yu Zhou, Mingyan Li, Hang Yu, Jinchi Yu, Mingchen Sun, Dadong WangSymmetry 2025 · DOI

TL;DR

This paper introduces a novel multi-path architecture for Manchu script recognition that effectively fuses spatial and spectral features using a Swin Transformer. By leveraging STFT for spectral analysis and a multi-branch CNN-Transformer approach, the model achieves state-of-the-art accuracy, particularly for degraded and multi-font Manchu documents, addressing significant challenges in recognizing this endangered language.

What Problem Does This Paper Address?

Recognizing Manchu script presents unique challenges due to its complex character variations, subtle differences between similar characters, and homographic polysemy. Traditional methods often rely on character segmentation, which can lead to errors and loss of semantic information. While convolutional neural networks (CNNs) have improved word recognition by treating words as images, they struggle with modeling long-range dependencies and capturing global context, especially in visually complex scripts like Manchu. This paper aims to overcome these limitations by developing a robust Manchu recognition system that can handle variations in writing styles, character positions, and image degradation.

Methods

The proposed method employs a multi-path architecture that combines spatial and spectral feature extraction.

  • Preprocessing: Manchu word images are first resized. Then, the Short-Time Fourier Transform (STFT) is applied to extract spectral features along both horizontal and vertical axes. This captures subtle variations and transitions within the Manchu strokes.
  • Multi-Path Encoding: The architecture uses three parallel branches:
  • A main encoder processes the original image to extract spatial features.
  • Two symmetric spectral encoding enhancement modules process the horizontal and vertical spectral features separately. These modules use CNNs with specific block designs that alternate between BatchNorm-after-Conv and Conv-after-BatchNorm patterns, and incorporate MaxPool for dimension reduction while preserving horizontal structure.
  • Feature Fusion: A Swin Transformer block is used to fuse the features from the two spectral branches. This fused spectral representation is then concatenated with the spatial features from the main encoder. The Swin Transformer is chosen for its ability to model long-range dependencies, crucial for Manchu's vertically connected script.
  • Decoding: A modified Transformer decoder is employed for sequence modeling. A key modification is the removal of positional encoding, as Manchu characters' forms are highly dependent on their position within a word, making standard positional encoding detrimental.

Datasets

The study utilizes two datasets for evaluation:

  • Dataset 1: A single-font dataset containing 130,917 Manchu words in the Abkai font.
  • Dataset 2: A multi-font dataset comprising 1570 words from 11 different fonts, derived from the KaikKi Manchu Dictionary.

In total, the study uses 12 fonts and 148,187 Manchu images. For training, approximately 80% of the multi-font dataset was used, with the remainder for testing. A subset of 90,000 words from the single-font dataset was split into a training set (50,000 words) and a test set (40,000 words).

Key Insights and Ideas

  • STFT for Spectral Features: Applying STFT to Manchu word images is effective in capturing subtle stroke variations and transitions, providing a robust basis for differentiation, especially for a cursive script.
  • Multi-Path Fusion is Crucial: Combining spatial features from the original image with spectral features from different orientations (horizontal and vertical) significantly enhances recognition accuracy, particularly for degraded images.
  • Swin Transformer for Long-Range Dependencies: The Swin Transformer's ability to model long-range dependencies is vital for Manchu script, where character shapes are heavily influenced by adjacent characters. This outperforms traditional CNNs in fusion tasks.
  • Positional Encoding Removal: Removing positional encoding from the Transformer decoder is a critical adaptation for Manchu, as its script exhibits strong positional allography, making standard positional encodings counterproductive.

Key Results

The proposed model achieves state-of-the-art performance on Manchu word recognition tasks:

  • Single-font test dataset: 99.81% accuracy.
  • Degraded single-font test dataset: 99.71% accuracy.
  • Multi-font test dataset: 99.75% accuracy.
  • Degraded multi-font test dataset: 99.69% accuracy.

The model also demonstrates superior performance compared to existing methods, outperforming the attention-based CRNN by 2.37 percentage points and Deep CNN by 0.95 percentage points on similar datasets. For multi-font recognition, it surpasses the CNN Transformers model by 4.0 percentage points. Ablation studies show that the 3-path Swin Transformer encoder and Swin Transformer blocks for fusion yield the highest accuracy, especially with degraded images.

Limitations and Future Work

  • Inference Time: The model's inference time is noted as relatively long (Tw = 0.1753 s, Tc = 0.0306 s).
  • Handwriting-Style Fonts: Accuracy for handwriting-style fonts remains relatively low, suggesting limited causal inference capabilities for such styles.
  • "Extreme" Cases: The model's accuracy on severely damaged or "extreme" cases of historical documents needs substantial improvement.

Future work will focus on enhancing the model's generalization to achieve higher recognition rates for damaged documents and exploring improvements for handwriting-style fonts.

Summary updated

BibTeX
@article{zhou2025research,
  author = {Yu Zhou and Mingyan Li and Hang Yu and Jinchi Yu and Mingchen Sun and Dadong Wang},
  title = {{Research on Multi-Path Feature Fusion Manchu Recognition Based on Swin Transformer}},
  journal = {Symmetry},
  year = {2025},
  doi = {10.3390/sym17091408},
}
Figure