When humans interact with the visual world, our language is inherently spatial. We easily understand instructions like "hand me the coffee mug on the far left" or "click the login button at the top right of the screen." For artificial intelligence models, however, bridging the gap between high-level semantic tokens and the precise geometry of an image is a profound challenge.
Standard Convolutional Neural Networks (CNNs) are designed to be translation-equivariant, meaning they process local features regardless of their absolute position (relying mostly on subtle boundary cues like zero-padding to infer location, as shown by Matthew Kowal et al. in Position, Padding and Predictions). Vision Transformers (ViTs) slice images into patches and use positional encodings, but these encodings are optimized to preserve sequence order rather than continuous geometric coordinates.
To overcome this limitation, a major research direction focuses on spatial coordinate grounding. By explicitly injecting coordinate systems into visual models—either internally through feature-level embeddings or externally via visual prompts—researchers have enabled models to reason about the physical layout of scenes with high precision.
Feature-Level Coordinate Injection
The most direct way to make a model coordinate-aware is to weave geometric data directly into its internal feature maps. Instead of forcing a network to infer coordinates purely from raw pixel patterns, coordinates are provided as an explicit parallel input.
In early foundational work on natural language object retrieval, Ronghang Hu et al. introduced the Spatial Context Recurrent ConvNet (Natural Language Object Retrieval), which explicitly appended a low-dimensional vector containing normalized bounding box coordinates, center points, and dimensions to the visual features. For dense pixel-level tasks like referring image segmentation, Chenxi Liu 0001 demonstrated in Recurrent Multimodal Interaction for Referring Image Segmentation that concatenating a normalized $[x, y]$ coordinate grid to visual feature vectors provides the necessary geometric anchor to resolve directional language.
Visual Feature Map (H x W x C) ───┐
├──► Fused Feature Map (H x W x (C + 2))
Normalized Coordinate Grid (H x W x 2) ┘
As multi-modal architectures shifted toward Transformers, this injection evolved from simple vector concatenation to high-dimensional geometric embeddings. In VL-BERT, Weijie Su et al. transformed region-of-interest coordinates using trigonometric functions, creating continuous geometry embeddings analogous to the sinusoidal positional encodings used for text tokens.
Why Feature Injection Works
By treating spatial coordinates as a first-class feature, the model does not have to waste parameter capacity trying to reverse-engineer pixel locations. When a text encoder processes a phrase like "on the left," the multi-modal attention layers can directly map that linguistic concept to the regions where the explicit coordinate features have low $x$-values. This creates a shared mathematical language where text, visual appearance, and geometry can align seamlessly.
Visual Coordinate Prompting
While feature-level injection is highly effective when training or fine-tuning models, it is impossible to apply to frozen, off-the-shelf Large Vision-Language Models (VLMs) like GPT-4V, which only accept raw images as input. To give these models spatial awareness, researchers developed visual coordinate prompting: overlaying coordinate grids, tick marks, or numbered bounding boxes directly onto the input pixels.
┌─────────────────────────┐ ┌─────────────────────────┐
│ [Object] │ │ 0,0───50───100 │
│ │ ───► │ │ │
│ │ │ 50 [Object] │
│ │ │ │ │
└─────────────────────────┘ └─────────────────────────┘
Raw Input Image Visual Grid Overlay
In Agent3D-Zero, Sha Zhang et al. demonstrated that superimposing Cartesian coordinate systems or grid lines onto 2D images significantly enhances a VLM's capacity for precise spatial reasoning and path planning. Similarly, Zihao Wan et al. showed in Bridging Vision, Language, and Mathematics that overlaying a normalized Cartesian grid acts as a physical anchor, allowing the model's vision encoder to map abstract numerical coordinate outputs directly to specific pixel locations.
This technique has become a cornerstone for general computer control (GCC) and digital agents. For instance, in Cradle, Weihao Tan et al. overlaid numbered bounding boxes and coordinate grids onto raw OS screenshots. This allowed the agent to bypass its lack of spatial precision and accurately map its semantic intentions (e.g., "click the settings icon") to precise pixel coordinates for mouse and keyboard interactions.
Why Visual Prompting Works
Visual prompting fundamentally changes the nature of the task for the VLM. Instead of forcing the model to perform a complex regression task (estimating abstract numerical coordinates from a raw image), the grid turns the task into a visual identification problem. The model simply reads the numbers and tick marks already present in its visual field, leveraging its highly optimized text-recognition and pattern-matching capabilities to achieve spatial precision.
Disambiguation and Relational Reasoning
Beyond simply locating a single object, coordinate awareness fundamentally alters how models handle scene layout and object relationships.
Disambiguating Dense Environments
In crowded scenes, visual appearance alone is often insufficient for identification. In A Fast and Accurate One-Stage Approach to Visual Grounding, Boqing Gong et al. demonstrated that concatenating normalized coordinate vectors allows models to resolve linguistic references to relative positions instantly. This geometric context is vital for tracking objects through total occlusions or dramatic appearance changes, as explored by Xiao Wang et al. in Towards More Flexible and Accurate Object Tracking With Natural Language.
Encoding Scene Structure
Knowing the absolute coordinates of objects allows a model to calculate their relative distances and orientations, unlocking deeper relational reasoning. In REVIVE, Yuanze Lin et al. showed that integrating bounding box coordinates directly into regional visual features provides a structural framework that drastically improves knowledge-based visual question answering.
This concept is also utilized in self-supervised learning: the framework introduced in Self-Supervised Learning for Visual Relationship Detection through Masked Bounding Box Reconstruction trains models by masking object features and forcing the architecture to reconstruct them using only the geometric embeddings of neighboring bounding boxes. This forces the model to learn how spatial layouts correlate with semantic relationships.
By anchoring visual features to explicit coordinates—whether through internal mathematical embeddings or external visual grids—modern vision-language models successfully bridge the gap between semantic understanding and geometric precision, transforming them from passive image observers into spatially aware agents.
Key Foundational Work
Segmentation from Natural Language Expressions (2016) — contributed 2 concept elements to this direction.
Visual Question Answering with Question Representation Update (QRU) (2016) — contributed 1 concept elements to this direction.
Natural Language Object Retrieval (2016) — contributed 1 concept elements to this direction.