OracleFusion: Assisting the Decipherment of Oracle Bone Script with Structurally Constrained Semantic Typography

Caoshuo Li, Zengmao Ding, Xiaobin Hu, Bang Li, Donghao Luo, AndyPian Wu, Chaoyang Wang, Chengjie Wang, Taisong Jin, SevenShu, Yunsheng Wu, Yongge Liu, Rongrong Ji

TL;DR

This paper introduces OracleFusion, a novel framework that uses multimodal large language models (MLLMs) and structural constraints to generate semantically rich vector fonts for Oracle Bone Script (OBS). This approach aims to aid experts in deciphering the vast number of undeciphered OBS characters by providing visually enhanced, structurally accurate interpretations.

What Problem Does This Paper Address?

Oracle Bone Script (OBS) is one of the earliest known forms of Chinese writing, offering invaluable insights into ancient civilizations. However, a significant portion of the approximately 4,500 discovered OBS characters remain undeciphered due to their complex structures and abstract imagery. Existing methods, like OBSD, can generate modern Chinese counterparts but lack explicit semantic analysis, limiting their interpretability and usefulness for decipherment. The challenge lies in systematically analyzing OBS glyph structures, understanding their component meanings, and reconstructing the scenes they represent while preserving the original glyph's integrity.

Methods

OracleFusion is a two-stage semantic typography framework:

  • Stage 1: Oracle Bone Semantics Understanding and Visual Grounding (OBSUG)
    • Leverages MLLMs with enhanced Spatial Awareness Reasoning (SAR) to analyze OBS glyph structures and localize key components (radicals).
  • SAR helps the MLLM understand the relative positioning of these components, forming a structured representation.
  • This stage aims to identify key structural components, their spatial relationships, and generate a semantic description of the glyph.
  • Stage 2: Structural Oracle Vector Fusion (SOVF)
    • A training-free framework that uses structural constraints to control the generation of semantically rich vectorized fonts.
  • Cross-Modal Attention mechanisms are employed to guide the generation process.
  • Glyph Structural Constraint ensures generated objects adhere to a specified layout, guiding the gradient of latent features.
  • Glyph Maintenance introduces Skeleton Structure (SKST) Loss to preserve the original glyph's structural fidelity by enforcing angle preservation and directional alignment.
  • The overall objective function combines losses for semantic representation (LLSDS), structural constraints (LGSDS), and glyph maintenance (LSKST, tone loss).

Datasets

  • RMOBS: A newly curated multimodal dataset comprising over 20,000 OBS samples across 900 deciphered characters. Each sample includes the oracle glyph, semantic concept, key components (radicals), and bounding box layout. This dataset is enriched with radical-level structural annotations.

Key Insights and Ideas

  • MLLMs for Structural Analysis: The paper demonstrates that MLLMs, particularly with SAR, can effectively analyze the complex structures of ancient scripts like OBS, going beyond simple image generation.
  • The Importance of Structural Constraints: Simply generating semantically relevant images isn't enough for decipherment. OracleFusion's SOVF stage highlights the critical need for structural constraints (both layout and skeleton preservation) to ensure the generated glyphs are faithful to the original and useful for linguistic analysis.
  • Bridging Symbolic and Computational Decipherment: OracleFusion acts as a bridge, translating abstract ancient symbols into visually interpretable vector fonts that can then be used by other advanced AI models for further analysis (e.g., colorization, texture addition).
  • RMOBS as a Valuable Benchmark: The creation of RMOBS, with its detailed radical-level annotations, provides a crucial resource for future research in OBS decipherment and related areas.

Key Results

  • Quantitative: OracleFusion achieved the highest scores in CLIPScore (semantic relevance) and the lowest distance (glyph maintenance) compared to baselines like ClipDraw and Word-As-Image. A user study also showed OracleFusion consistently outperformed other methods in semantics, visual appeal, and glyph maintenance.
  • Qualitative: OracleFusion successfully generated semantically rich and structurally accurate vector fonts for both deciphered and undeciphered OBS characters. It accurately captured complex meanings and structural components, providing plausible interpretations for unseen characters that could aid expert analysis. The generated glyphs were visually appealing and maintained the integrity of the original script.

Limitations and Future Work

  • Limitations: The paper acknowledges that while the multi-turn approach in OBSUG improves accuracy, it slightly reduces mIoU. The balance between shape preservation and semantic expression is crucial, and the chosen parameter for SKST loss (β=0.5) is an optimal point found through experimentation.
  • Future Work: The authors suggest expanding the multimodal dataset, enhancing OracleFusion's capabilities, and exploring broader applications in ancient script analysis. They also propose integrating with other frameworks for post-processing, such as adding color and texture to the generated glyphs.

Summary updated

Abstract

As one of the earliest ancient languages, Oracle Bone Script (OBS) encapsulates the cultural records and intellectual expressions of ancient civilizations. Despite the discovery of approximately 4,500 OBS characters, only about 1,600 have been deciphered. The remaining undeciphered ones, with their complex structure and abstract imagery, pose significant challenges for interpretation. To address these challenges, this paper proposes a novel two-stage semantic typography framework, named OracleFusion. In the first stage, this approach leverages the Multimodal Large Language Model (MLLM) with enhanced Spatial Awareness Reasoning (SAR) to analyze the glyph structure of the OBS character and perform visual localization of key components. In the second stage, we introduce Oracle Structural Vector Fusion (OSVF), incorporating glyph structure constraints and glyph maintenance constraints to ensure the accurate generation of semantically enriched vector fonts. This approach preserves the objective integrity of the glyph structure, offering visually enhanced representations that assist experts in deciphering OBS. Extensive qualitative and quantitative experiments demonstrate that OracleFusion outperforms state-of-the-art baseline models in terms of semantics, visual appeal, and glyph maintenance, significantly enhancing both readability and aesthetic quality. Furthermore, OracleFusion provides expert-like insights on unseen oracle characters, making it a valuable tool for advancing the decipherment of OBS.
BibTeX
@inproceedings{li2025oraclefusion,
  author = {Caoshuo Li and Zengmao Ding and Xiaobin Hu and Bang Li and Donghao Luo and AndyPian Wu and Chaoyang Wang and Chengjie Wang and Taisong Jin and SevenShu and Yunsheng Wu and Yongge Liu and Rongrong Ji},
  title = {{OracleFusion: Assisting the Decipherment of Oracle Bone Script with Structurally Constrained Semantic Typography}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2025},
}
Figure