ArabGeoText: Tool-Augmented Vision-Language Models for Arabic Geospatial Text Spotting

Yakoub Bazi, Sarah Alsaleh, Mohamad M. Al Rahhal, Mohamed A. Mekhtiche, Mansour ZuairarXiv 2026 · arXiv

TL;DR

The authors introduce ARABGEOTEXT, the first Arabic-centric benchmark for geospatial text spotting, featuring paired map and satellite-overlay views. They demonstrate that while large Vision-Language Models (VLMs) are powerful, a "tool-augmented" approach—using a specialized YOLOv10 detector to feed rectified crops to a LoRA-adapted Qwen3.5-4B reader—significantly outperforms direct end-to-end spotting.

What Problem Does This Paper Address?

Geospatial text spotting is the task of automatically locating and transcribing labels (like street names and landmarks) from raster maps. This is notoriously difficult because map text is often tiny, densely packed, and oriented at arbitrary angles.

For Arabic, the challenge is even steeper. The script's cursive nature, where letter forms change based on their position, and the reliance on small dots (diacritics) for character differentiation, makes it highly sensitive to low resolution and noise. While Arabic OCR has matured for documents and natural scenes, geospatial data has remained a major gap. Furthermore, existing models often struggle to generalize when text is overlaid on complex backgrounds, such as remote sensing (RS) satellite imagery.

Methods

The researchers compared two primary paradigms for solving this task:

  1. Direct VLM Spotting: A single Qwen3.5-4B model is prompted to output a structured list of all polygons and their corresponding transcriptions directly from a full 1024x1024 image.
  2. Tool-Augmented VLM Spotting: This pipeline decouples localization from recognition:
  3. Localization: A specialized YOLOv10 instance segmentation model identifies text regions.
  4. Crop Preparation: Detected regions are extracted, rectified using a projective transform to "un-rotate" the text, and upscaled to a standard size.
  5. Recognition: The processed crops are fed to a Qwen3.5-4B reader. The authors evaluated both a zero-shot version and one fine-tuned using Low-Rank Adaptation (LoRA) on the specific vocabulary and styles of the benchmark.
Fig. 3. Three paradigms for VLM-based geospatial text spotting: (a) direct spotting, (b) tool-augmented spotting with zero-shot recognition, and (c) tool- augmented spotting with adapted recognition.

Datasets

The paper introduces the ARABGEOTEXT dataset, which consists of 970 geographic locations across Riyadh and Jeddah, Saudi Arabia.

  • Paired Views: Each location includes a standard OpenStreetMap (OSM) rendering and a composite (RS+OSM) where the map is overlaid on satellite imagery. This allows researchers to test how background clutter affects performance.
  • Annotations: 17,875 manually annotated text instances per view type, including Arabic, Latin, and mixed-script labels.
  • Splits: Riyadh data is used for training and "in-city" testing (Test-I), while Jeddah is reserved for "city-disjoint" testing (Test-II) to evaluate geographic generalization.

Key Insights and Ideas

  • The "Full-Image" Bottleneck: VLMs struggle to attend to dozens of tiny, rotated text instances simultaneously. By delegating the "where" to YOLO and the "what" to the VLM, the system achieves much higher precision.
  • Rectification Matters: Feeding the VLM a "straightened" and upscaled crop (as seen in Figure 4) significantly lowers the character error rate (CER) compared to asking the model to read rotated text in a cluttered scene.
  • Localization Interference: Interestingly, the authors found that asking a VLM to perform joint transcription and localization actually degrades its localization accuracy compared to a VLM trained only to detect regions.
  • Background Robustness: The tool-augmented pipeline showed remarkably consistent performance across both clean OSM maps and noisy RS+OSM composites, suggesting that specialized detectors are better at filtering out satellite background noise than general-purpose VLMs.
Fig. 4. Crop preparation: (a) text region detected by YOLO26, (b) its axis-aligned crop, and (c) the rectified and upscaled crop supplied to the VLM reader.

Key Results

The quantitative gap between the direct and tool-augmented approaches is stark:

  • Detection: YOLOv10 achieved an H-mean (a balance of precision and recall) of 88.2% on Riyadh OSM, while the direct VLM managed only 34.5%.
  • End-to-End Spotting: The tool-augmented pipeline with a LoRA-adapted reader reached an E2E H-mean of 74.4% (Riyadh OSM). The direct VLM approach lagged far behind at 12.8%.
  • Generalization: When moving to the unseen city of Jeddah (Test-II), performance dropped (E2E H-mean of 57.3%), highlighting the difficulty of handling new vocabularies and smaller text sizes in different urban layouts.
Fig. 5. Localization examples. Rows: Test-I OSM, Test-I RS+OSM, Test-II OSM, and Test-II RS+OSM. Columns: (a) ground truth, (b) adapted direct Qwen3.5-4B, and (c) YOLO26. Reference polygons are magenta; predictions are blue.

Limitations and Future Work

While the tool-augmented approach is superior in accuracy, it is computationally heavier, requiring multiple VLM calls (one per detected text crop). The authors noted that using a smaller 0.8B parameter model for the reader can speed up inference by 1.7x, though with a slight hit to accuracy.

Future work aims to improve efficiency and explore better ways to generalize to new cities where the vocabulary and map styles might differ significantly from the training data. The authors also suggest expanding the benchmark to include more diverse geographic regions.

Summary updated

Abstract

Geospatial text spotting involves locating and transcribing geographic labels, such as road names, district names, and landmarks, in raster maps. These labels are often small, densely arranged, and arbitrarily oriented. Although Arabic text recognition has been studied extensively in documents and natural scenes, Arabic geospatial text spotting remains underexplored. We introduce ARABGEOTEXT, an Arabic-centric benchmark covering 970 geographic locations in Riyadh and Jeddah, with 17,875 manually annotated text instances per view type. For each location, we pair an OpenStreetMap (OSM) rendering with a composite created by overlaying that rendering on the corresponding remote sensing (RS) imagery. The paired views preserve label content and pixel geometry, allowing evaluation across different backgrounds. Geographic partitions support in-city and city-disjoint evaluation of detection, recognition, and end-to-end spotting. Experiments with direct Qwen3.5-4B spotting show difficulties in recovering all labels within a tile. We therefore propose a tool-augmented vision-language framework that combines YOLO26 localization and crop rectification with Qwen3.5-4B recognition adapted through low-rank adaptation (LoRA). The framework outperforms direct LoRA-adapted Qwen3.5-4B spotting across both image views and test cities. With localization held fixed, reader adaptation further improves recognition over the pretrained zero-shot reader. The dataset and code will be made publicly available at https://github.com/yakoubbazi/ArabGeoText.
BibTeX
@misc{bazi2026arabgeotext,
  author = {Yakoub Bazi and Sarah Alsaleh and Mohamad M. Al Rahhal and Mohamed A. Mekhtiche and Mansour Zuair},
  title = {{ArabGeoText: Tool-Augmented Vision-Language Models for Arabic Geospatial Text Spotting}},
  year = {2026},
  eprint = {2611.03339},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.03339},
}
Figure