Traditionally, computer vision and natural language processing spoke entirely different languages. Vision models predicted continuous numerical values for bounding boxes via regression heads, while language models predicted discrete tokens from a fixed vocabulary.
The paradigm of textual coordinate representation bridges this gap by encoding spatial coordinates directly as text tokens. Instead of building complex multi-modal fusion layers or auxiliary task-specific heads, this approach treats spatial localization as a standard sequence-to-sequence problem. By formatting coordinates as text, models can seamlessly interleave spatial reasoning with natural language generation.
The Mechanics of Textual Coordinates
To process spatial data within a language model, continuous physical space (such as image pixels or 3D coordinates) must be mapped into the model's textual vocabulary. Researchers have converged on two primary strategies to achieve this:
1. Quantization into Location Tokens
In this approach, continuous coordinates are normalized (typically between $0$ and $1$) and divided into discrete bins. Each bin corresponds to a unique "location token" added to the model's vocabulary.
For example, Zhengyuan Yang et al. introduced UniTAB, which quantizes bounding box coordinates into discrete tokens and interleaves them directly within text sequences using structural markers. Similarly, Zhiliang Peng et al. utilized location tokens in Kosmos-2 to explicitly link text spans to spatial coordinates within a unified sequence, allowing the model to read and output coordinates as if they were standard words.
2. Plain Text Decimal Strings
Instead of expanding the vocabulary with specialized location tokens, coordinates can be written out as standard numbers and decimals (e.g., "0.45, 0.62").
Dongkai Wang et al. demonstrated in LocLLM that representing coordinates as standard decimal strings allows for sub-pixel localization accuracy without modifying the underlying token embeddings or model architecture. This approach leverages the model's pre-existing understanding of numbers and punctuation. Zhaowei Li et al. adopted a similar strategy in GroundingGPT, showing that large language models can use their inherent numerical reasoning capabilities to handle spatial and temporal localization tasks across text, images, and video.
Why It Works: Leveraging Sequence Priors
Framing spatial coordinates as text works because it allows models to treat visual grounding as a syntactic property of the sequence.
When a model learns to predict a bounding box inline—such as <obj> the blue car </obj> [x1, y1, x2, y2]—the relationship between the object description and its physical location is learned purely through autoregressive next-token prediction. The model's self-attention mechanism naturally learns to anchor specific text phrases to their corresponding spatial coordinates.
Surprisingly, this textual abstraction can even benefit models that do not see the raw pixels at all. Ander Salaberria et al. demonstrated in their work that text-only language models pre-trained on object labels and discretized coordinates can outperform complex multimodal architectures at understanding spatial relations like "left of" or "inside." By abstracting scenes into textual coordinates, the model bypasses raw pixel processing and focuses entirely on the underlying geometric logic.
Architectural Simplicity: Eliminating Specialized Heads
In traditional object detection or visual grounding architectures, models rely on specialized components like Region-of-Interest (RoI) Align layers, anchor boxes, and custom regression losses (such as GIoU loss). Textual coordinate representation eliminates this architectural overhead entirely.
Traditional VLM:
[Image] ──> [Vision Encoder] ──> [Language Model] ──> [Text Output]
└──> [Regression Head] ──> [Bounding Box]
Textual Coordinate VLM:
[Image] ──> [Vision Encoder] ──> [Language Model] ──> [Unified Text & Box Sequence]
By unifying the output space into a single text stream, the entire system can be optimized using standard cross-entropy loss.
- Kartik Kuckreja et al. applied this to remote sensing in GeoChat, showing that a single language decoder can handle region-specific dialogue and visual grounding without specialized non-textual output heads.
- Yang Zhan et al. reinforced this with SkyEyeGPT, demonstrating that standard token prediction mechanisms are sufficient for complex geometric reasoning.
- Wenfang Sun et al. noted in RegionReasoner that treating visual grounding as a language modeling objective makes it straightforward to apply standard reinforcement learning techniques for optimization, which is traditionally difficult when dealing with separate regression heads.
Generalization Across Domains
Because coordinates are represented as standard text sequences, this technique generalizes across diverse domains that require an understanding of physical space:
- Robotics and Embodied AI: In the OCI framework, Junjie Wen et al. wove spatial coordinates directly into language prompts, allowing policy networks to process positional information as semantic context. For 3D building tasks in virtual environments, Xin Deng demonstrated in their paper that representing 3D grid coordinates as natural language strings allows sequence-to-sequence Transformers to successfully perform grounded collaborative reasoning.
- Autonomous Driving: Jiageng Mao et al. developed GPT-Driver, which converts heterogeneous sensor data—such as object coordinates and velocities—into natural language descriptions to formulate motion planning as a language modeling problem. To reason about semantic road rules, Xu Cao et al. similarly tokenized high-definition map data (like lane markings and intersection boundaries) into structured text strings for MAPLM.
- GUI Agents: Kanzhi Cheng et al. created SeeClick, a visual GUI agent that maps text instructions directly to plain text screen coordinates. This allows the agent to operate purely on screenshots, bypassing the need to parse complex underlying HTML or DOM structures.
Key Foundational Work
GPT-Driver: Learning to Drive with GPT (2023) — contributed 1 concept elements to this direction.
Mapping Language Models to Grounded Conceptual Spaces (2022) — contributed 1 concept elements to this direction.
UniTAB: Unifying Text and Box Outputs for Grounded Vision-Language Modeling (2022) — contributed 1 concept elements to this direction.