Modality Adaptation Representation Learning for Historical Document Image Retrieval

Jing Zhang, Hongxi Wei, Qing ZhangData 2025 · DOI

TL;DR

Researchers have developed MARK, a new framework that significantly improves how we search for specific words in historical document images. By combining the vision-language power of CLIP with the flexible mapping of Kolmogorov-Arnold Networks (KAN) and character-level encoding (PHOC), MARK bridges the "modality gap" between text and images, setting new performance records on challenging Mongolian and Manchu datasets.

What Problem Does This Paper Address?

The paper tackles Historical Document Image Retrieval (HDIR), specifically a task called "word spotting." Imagine you have thousands of scanned pages of ancient manuscripts and you want to find every instance of a specific word. You can do this via Query-by-Example (QBE), where you provide an image of the word, or Query-by-String (QBS), where you type the word.

QBS is particularly difficult because it requires the computer to understand that a digital string of characters (text) and a noisy, handwritten, or woodblock-printed snippet (image) represent the same thing. This is the "modality gap." Furthermore, historical languages like Mongolian are "agglutinative"—they create words by stacking suffixes onto roots—leading to a virtually infinite vocabulary. This creates an Out-of-Vocabulary (OOV) problem where the model must find words it never saw during training.

Methods

The authors propose the Modality Adaptation Representation Learning (MARK) framework. Its architecture rests on three pillars:

  1. Backbone (MetaCLIP): They use MetaCLIP to extract initial features. While CLIP is excellent at general image-text alignment, it often misses the fine-grained details needed to distinguish between two very similar-looking handwritten words.
  2. Bi-directional Modality Adaptation: Instead of just forcing image and text features into the same space, MARK uses two modules:
  3. Image-to-Text Adaptation (ITA): Translates image features into the textual space.
  4. Text-to-Image Adaptation (PTIA): Translates text features into the image space.
  5. Kolmogorov-Arnold Networks (KAN): Unlike traditional Multi-Layer Perceptrons (MLPs) that use fixed activation functions on nodes, KANs use learnable activation functions on the "edges" (connections). This makes them much better at learning the complex, non-linear mappings required to bridge the gap between a visual word and its digital text counterpart.
  6. PHOC Integration: To help the model "see" individual characters, they incorporate Pyramid Histograms of Characters (PHOC). PHOC represents a word by recording which characters appear in which spatial segments (e.g., "Is the letter 'A' in the first half of the word?").

Datasets

The model was evaluated on three distinct historical datasets:

  • Kanjur: 20,948 Mongolian word images from woodblock prints.
  • Geser: A larger Mongolian dataset with both woodblock and manuscript styles, featuring specific "OOV splits" (20% to 80%) to test how well the model handles unseen words.
  • Manchu: A recently released dataset of 2,428 unique Manchu words, used to prove the model's ability to generalize across different (but linguistically related) scripts.

Key Insights and Ideas

  • CLIP is "Near-Sighted" for Text: The authors found that while CLIP is great at recognizing a "dog" vs. a "cat," it struggles with fine-grained text (e.g., "word" vs. "work"). Integrating PHOC acts like a pair of reading glasses, forcing the model to pay attention to character-level details.
  • The Power of KAN: Replacing standard MLPs with KANs provided more stable training and better alignment. The KAN-based "Gating Mechanism" in their attention blocks allowed the model to adaptively focus on the most relevant dimensions of the feature vectors.
  • Bi-directional is Better: By adapting in both directions (Image $\to$ Text and Text $\to$ Image), the model captures both the shared semantic meaning and the unique complementary information present in each modality.

Key Results

MARK established a new State-of-the-Art (SOTA) across the board:

  • Kanjur Dataset: Achieved a 99.16% mAP (mean Average Precision) for QBE and 97.39% mAP for QBS. For context, vanilla CLIP only managed 76.48% on the QBS task.
  • Geser Dataset: Even when 80% of the test words were "unseen" (OOV) during training, MARK maintained high performance, significantly outperforming previous specialized models like HENet.
  • Manchu Dataset: Achieved 98.29% mAP for QBS, proving that the framework isn't just a "one-trick pony" for Mongolian but works for other low-resource, morphologically rich languages.

Limitations and Future Work

While MARK is highly effective, the paper notes a few areas for growth:

  • Text-Centric Focus: The current work is highly optimized for word-level retrieval. Future work could extend this to full-page document understanding or more complex visual question-answering in historical contexts.
  • Computational Efficiency: While they used "FastKAN" to improve speed, KANs are generally more complex than MLPs. Scaling this to massive, million-image libraries might require further optimization.
  • Segmentation Reliance: The model currently operates on pre-segmented word images. Integrating this into a "segmentation-free" pipeline (where the model finds words directly on a full page) would be a logical next step.

Summary updated

Abstract

Word spotting is a fundamental task in Historical Document Image Retrieval (HDIR), categorized into query-by-example (QBE) and query-by-string (QBS) based on the query modality. QBS word spotting, as a cross-modal retrieval task, poses a significant challenge due to the substantial modality gap between image and text representations. Existing methods typically embed both image and text features into a common space and directly constrain their alignment, overlooking the substantial distribution gaps between modalities. Moreover, out-of-vocabulary (OOV) words present an additional challenge, requiring the model to possess strong generalization capability. To address these challenges, we propose a novel bi-directional modality adaptation representation learning framework, termed MARK. Specifically, the model adopts CLIP as the backbone to extract image and text features, providing a strong baseline for cross-modal representations. A bi-directional modality adaptation mechanism based on the Kolmogorov-Arnold network (KAN) is designed to capture both shared and complementary information between image and text representations. Furthermore, Pyramid Histograms of Characters (PHOC) is introduced to enhance fine-grained text representation and overcome the limitations of CLIP’s text encoder. Extensive experiments on Monglian and Manchu benchmark datasets, including Kanjur, Geser and Manchu, demonstrate that MARK establishes a new state-of-the-art in HDIR. Our code is available at <uri xmlns:xlink="http://www.w3.org/1999/xlink" xlink:href="https://github.com/cszjing/MARK">https://github.com/cszjing/MARK</uri>.
BibTeX
@inproceedings{zhang2025modality,
  author = {Jing Zhang and Hongxi Wei and Qing Zhang},
  title = {{Modality Adaptation Representation Learning for Historical Document Image Retrieval}},
  booktitle = {Data},
  year = {2025},
  doi = {10.3724/2096-7004.di.2025.0111},
}
Figure