TL;DR
This paper introduces OracleFusion, a novel framework that uses multimodal large language models (MLLMs) and structural constraints to generate semantically rich vector fonts for Oracle Bone Script (OBS). This approach aims to aid experts in deciphering the vast number of undeciphered OBS characters by providing visually enhanced, structurally accurate interpretations.
What Problem Does This Paper Address?
Oracle Bone Script (OBS) is one of the earliest known forms of Chinese writing, offering invaluable insights into ancient civilizations. However, a significant portion of the approximately 4,500 discovered OBS characters remain undeciphered due to their complex structures and abstract imagery. Existing methods, like OBSD, can generate modern Chinese counterparts but lack explicit semantic analysis, limiting their interpretability and usefulness for decipherment. The challenge lies in systematically analyzing OBS glyph structures, understanding their component meanings, and reconstructing the scenes they represent while preserving the original glyph's integrity.
Methods
OracleFusion is a two-stage semantic typography framework:
- Stage 1: Oracle Bone Semantics Understanding and Visual Grounding (OBSUG)
- Leverages MLLMs with enhanced Spatial Awareness Reasoning (SAR) to analyze OBS glyph structures and localize key components (radicals).
- SAR helps the MLLM understand the relative positioning of these components, forming a structured representation.
- This stage aims to identify key structural components, their spatial relationships, and generate a semantic description of the glyph.
- Stage 2: Structural Oracle Vector Fusion (SOVF)
- A training-free framework that uses structural constraints to control the generation of semantically rich vectorized fonts.
- Cross-Modal Attention mechanisms are employed to guide the generation process.
- Glyph Structural Constraint ensures generated objects adhere to a specified layout, guiding the gradient of latent features.
- Glyph Maintenance introduces Skeleton Structure (SKST) Loss to preserve the original glyph's structural fidelity by enforcing angle preservation and directional alignment.
- The overall objective function combines losses for semantic representation (LLSDS), structural constraints (LGSDS), and glyph maintenance (LSKST, tone loss).
Datasets
- RMOBS: A newly curated multimodal dataset comprising over 20,000 OBS samples across 900 deciphered characters. Each sample includes the oracle glyph, semantic concept, key components (radicals), and bounding box layout. This dataset is enriched with radical-level structural annotations.
Key Insights and Ideas
- MLLMs for Structural Analysis: The paper demonstrates that MLLMs, particularly with SAR, can effectively analyze the complex structures of ancient scripts like OBS, going beyond simple image generation.
- The Importance of Structural Constraints: Simply generating semantically relevant images isn't enough for decipherment. OracleFusion's SOVF stage highlights the critical need for structural constraints (both layout and skeleton preservation) to ensure the generated glyphs are faithful to the original and useful for linguistic analysis.
- Bridging Symbolic and Computational Decipherment: OracleFusion acts as a bridge, translating abstract ancient symbols into visually interpretable vector fonts that can then be used by other advanced AI models for further analysis (e.g., colorization, texture addition).
- RMOBS as a Valuable Benchmark: The creation of RMOBS, with its detailed radical-level annotations, provides a crucial resource for future research in OBS decipherment and related areas.
Key Results
- Quantitative: OracleFusion achieved the highest scores in CLIPScore (semantic relevance) and the lowest distance (glyph maintenance) compared to baselines like ClipDraw and Word-As-Image. A user study also showed OracleFusion consistently outperformed other methods in semantics, visual appeal, and glyph maintenance.
- Qualitative: OracleFusion successfully generated semantically rich and structurally accurate vector fonts for both deciphered and undeciphered OBS characters. It accurately captured complex meanings and structural components, providing plausible interpretations for unseen characters that could aid expert analysis. The generated glyphs were visually appealing and maintained the integrity of the original script.
Limitations and Future Work
- Limitations: The paper acknowledges that while the multi-turn approach in OBSUG improves accuracy, it slightly reduces mIoU. The balance between shape preservation and semantic expression is crucial, and the chosen parameter for SKST loss (β=0.5) is an optimal point found through experimentation.
- Future Work: The authors suggest expanding the multimodal dataset, enhancing OracleFusion's capabilities, and exploring broader applications in ancient script analysis. They also propose integrating with other frameworks for post-processing, such as adding color and texture to the generated glyphs.
Summary updated