The Visual Script Gap in Multimodal Models

This direction is no longer present in the current build. Its last version stays readable and citable here.

Version 1 of 119 papers31 conceptsfrozen 2026-10-04

Current Large Multimodal Models (LMMs) like GPT-4V and Gemini exhibit a striking paradox: they can discuss complex topics in dozens of languages, yet they often fail to read a simple "Stop" sign if it is written in a non-Latin script. This "visual script gap" reveals that a model’s linguistic proficiency in text does not automatically translate to its visual perception.

The OCR Bottleneck

For a multimodal model to reason about an image—whether it’s a social media meme, a financial chart, or a restaurant menu—it must first perform Optical Character Recognition (OCR). Research by Hanjia Lyu and colleagues in GPT-4V(ision) as A Social Media Analysis Engine demonstrates that while these models excel at identifying "hateful memes" in English, their performance degrades sharply when encountering non-Latin scripts like Chinese, Japanese, or Korean.

This isn't just a minor error in transcription; it is a fundamental failure of visual grounding. As Yang Wu notes in An Early Evaluation of GPT-4V(ision), a model might have near-perfect accuracy in English text extraction while failing to recognize basic characters in other languages. This suggests that the visual encoders—the "eyes" of the model—are often trained on predominantly Western datasets, leaving them "blind" to the orthographic nuances of other writing systems.

Why Generalization Fails

One might assume that because a model understands the concept of a language in its text-only mode, it would easily learn to recognize its visual form. However, evidence from MOTBench by Zhanglin Wu and others shows that proficiency in one script does not reliably predict performance in another.

The difficulty lies in the unique structural properties of different scripts:

  • Logographic Complexity: Traditional Chinese characters involve intricate stroke patterns that standard vision encoders often struggle to resolve, a problem highlighted by Chia-Sheng Liu in the development of the Breeze 2 models.
  • Cursive and Diacritics: Arabic script features cursive connections and complex diacritics. Sara Ghaboura and the team behind CAMEL-Bench found that these features pose significant hurdles for models optimized for the discrete, blocky nature of Latin text.
  • Ligatures and Orientation: Indic scripts use complex ligatures, while some minority languages use right-to-left orientations. Sha Jiu argues in TVQACML that these variations require specialized architectural handling that general-purpose models currently lack.

The "Sovereign" AI Solution

To bridge this gap, researchers are moving away from "one-size-fits-all" global models toward "sovereign" or localized AI. Geewook Kim and Sanghee Park demonstrated in KoNET that models specifically tuned for a local language and culture can significantly outperform much larger, closed-source models.

A key diagnostic tool in this research is comparing native multimodal performance against "OCR-assisted" performance. In many cases, a model performs better if an external, specialized OCR tool extracts the text first and feeds it to the model as a string. This confirms that the bottleneck isn't the model's "brain" (its reasoning capability), but its "eyes" (the vision encoder).

Scaling Multilingual Vision

The path forward involves massive, diverse datasets that treat visual text as a primary citizen. Xiang Yue and Yueqi Song introduced Pangea, which uses a 6-million-sample multilingual dataset to force the model to align visual features with 39 different languages. Similarly, Yichen Xu showed in POLYCHARTQA that targeted instruction tuning on multilingual charts can boost accuracy by over 90%, proving that the visual script gap is not an insurmountable architectural flaw, but a data-driven one.

By explicitly training models to recognize that a specific height of a bar in a chart corresponds to a label written in Hindi or Arabic, researchers are finally enabling AI to move beyond its English-centric origins and achieve true global visual literacy.

Papers in this version

History & citationVersion 1 · 2026-10-04

No longer present in the current build · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_the_visual_script_gap_in_multimodal_mode_3e106678,
  title = {The Visual Script Gap in Multimodal Models},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/the-visual-script-gap-in-multimodal-models/rdr_3e106678b98c4b33b6ec40d9fc549260},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-18},
}

This exact version (version 1, frozen text):

@misc{lacuna_the_visual_script_gap_in_multimodal_mode_3e106678_v1,
  title = {The Visual Script Gap in Multimodal Models},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/the-visual-script-gap-in-multimodal-models/rdr_3e106678b98c4b33b6ec40d9fc549260/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-18},
}