When we train a neural network to understand an image or a 3D scene, we traditionally ask it to do two fundamentally different things at once: predict discrete categories (like words or object classes) and regress continuous numbers (like bounding box coordinates or 3D positions). This split requires a hybrid architecture. We end up attaching specialized regression heads to our networks and balancing a delicate cocktail of loss functions—combining cross-entropy for text with $L_1$, $L_2$, or Intersection-over-Union (IoU) losses for geometry.
Discrete spatial tokenization eliminates this architectural split. Instead of treating space as a continuous field for regression, it slices space into a grid of discrete bins and assigns each bin a unique token in the model's vocabulary. By translating coordinates into words, spatial reasoning becomes a standard sequence-to-sequence problem.
This approach allows a vanilla Transformer to process text, pixels, and geometry through a single unified interface, optimizing everything with a single cross-entropy loss.
The Mechanism: Slicing Space into Words
To understand how continuous coordinates become discrete tokens, consider a standard 2D image. In a traditional object detection setup, a bounding box is represented by four continuous numbers: $[x_{min}, y_{min}, x_{max}, y_{max}]$, usually normalized between $0.0$ and $1.0$.
Under a spatial tokenization framework, we divide the image into a fixed-resolution grid—for example, a $1000 \times 1000$ grid. A coordinate like $0.384$ simply maps to the integer bin $384$. We then extend the Transformer's vocabulary with a set of dedicated location tokens (e.g., <loc_0> through <loc_999>). A bounding box that once required a specialized regression head is now generated as a sequence of four discrete text tokens: <loc_384> <loc_512> <loc_420> <loc_600>.
Continuous Space (0.0 to 1.0)
[ 0.384, 0.512, 0.420, 0.600 ]
│ │ │ │ (Quantize to 1000 bins)
▼ ▼ ▼ ▼
[ 384, 512, 420, 600 ]
│ │ │ │ (Map to Vocabulary)
▼ ▼ ▼ ▼
<loc_384> <loc_512> <loc_420> <loc_600> --> Processed just like words
As demonstrated by Ting Chen et al. in A Unified Sequence Interface for Vision Tasks, this simple quantization allows a single model to seamlessly interleave text and spatial coordinates. For instance, a visual grounding task can be framed as a pure translation problem:
"Find the dog" $\rightarrow$ "<obj> dog <loc_384> <loc_512> <loc_420> <loc_600>"
This serialization strategy has been successfully scaled across diverse domains:
- Unified Vision-Language: In OFA, Peng Wang et al. used a shared global vocabulary for text, image codes, and bounding boxes, proving that a single sequence-to-sequence model can handle object detection, captioning, and visual grounding without task-specific heads. Similarly, Chaoyang Zhu et al. demonstrated in SeqTR that serializing bounding boxes and segmentation masks into discrete coordinate sequences matches or exceeds the performance of complex, hand-crafted geometric architectures.
- Adaptive Quantization: Rather than using uniform grids, Zecheng Tang et al. introduced LayoutNUWA, which uses $k$-means clustering to adaptively quantize continuous layout coordinates, preserving high spatial precision while minimizing vocabulary size.
- Alternative Coordinate Systems: When moving to 3D spaces—such as molecular geometries—Cartesian $XYZ$ coordinates can cause the token vocabulary to explode. To solve this, Xiner Li et al. developed Geo2Seq, which maps 3D coordinates into bounded spherical coordinate components (angles and distances). This naturally bounds the token space, allowing standard language models to generate 3D molecular structures up to 100x faster than traditional diffusion models.
Why It Works: The Power of Classification Over Regression
Slicing continuous space into arbitrary integer bins might feel like it should degrade performance due to quantization errors. Yet, treating coordinates as discrete tokens often works better than continuous regression. There are three core intuitive reasons for this.
1. Handling Multi-Modality and Avoiding the "Average" Trap
Continuous regression losses like Mean Squared Error ($L_2$) force a model to output a single expected value. If a model is uncertain whether an object belongs on the far left ($x=0.1$) or the far right ($x=0.9$), an $L_2$ loss forces it to minimize its penalty by predicting the average: the exact middle ($x=0.5$). In physical space, this middle ground is often empty air or a collision.
By contrast, cross-entropy loss over discrete tokens models a categorical probability distribution. The model is free to assign high probabilities to <loc_100> and <loc_900> while assigning zero probability to <loc_500>.
Charlie Nash et al. leveraged this property in PolyGen to generate clean 3D meshes. Because the model outputs categorical distributions over quantized vertex positions, it can express sharp, multi-modal uncertainties without producing the "blurry" or averaged geometries typical of continuous regression models.
2. Exploiting Autoregressive Dependencies
When a model regresses a bounding box via a parallel head, it often predicts the four corners independently. It has no explicit mechanism to ensure that $x_{max}$ is logically larger than $x_{min}$ other than the raw features provided by the backbone.
When coordinates are tokenized and predicted autoregressively, the Transformer conditions each coordinate on every coordinate that came before it. If the model has already generated <loc_384> for $x_{min}$, the causal attention mechanism naturally restricts the probability distribution for $x_{max}$ to tokens greater than $384$.
In VAST, Yongshuai Huang et al. showed that this sequential modeling captures tight internal dependencies between coordinate axes, significantly improving structural precision in dense layout tasks like table structure recognition.
3. Bypassing Sub-word Tokenizer Fragmentation
If you pass raw numbers like 384 as text to an off-the-shelf Large Language Model (LLM), a standard sub-word tokenizer (like Byte-Pair Encoding) will fragment the string into arbitrary pieces based on text frequency (e.g., 38 and 4). This fragmentation completely destroys the numerical and spatial relationships between numbers.
As Jian Chen et al. demonstrated in TextLap, mapping coordinates to dedicated, atomic vocabulary tokens prevents this fragmentation. The model treats each geometric position as an indivisible unit, preserving numerical consistency and unlocking the model's latent spatial reasoning capabilities.
Unlocking Latent Spatial Priors
Perhaps the most powerful aspect of spatial tokenization is that it allows us to feed physical space directly into the native vocabulary of pre-trained language models. Once space is formatted as text, LLMs can apply their massive structural and logical priors to physical environments.
Layout Planning and Graphics
Instead of calculating pixel-perfect coordinates from scratch, models can treat graphic design as a code-generation or sequence-completion task. For example, Jaemin Cho et al. developed Visual Programming, which uses an LLM to generate discrete layout sequences that serve as intermediate blueprints for text-to-image generation. Because the layout is generated within a standardized grid vocabulary, the model can easily reason about relative object scales and positioning before any pixels are rendered.
Robotics and 3D Scene Synthesis
This linguistic bridging extends directly to physical control and 3D reconstruction. In RoboPrompt, Yida Yin et al. demonstrated that continuous 6-Degree-of-Freedom (6-DoF) robot poses can be partitioned into discrete text-based bins. By doing so, an off-the-shelf, text-only language model can predict precise robotic manipulation trajectories purely through in-context learning, treating physical control as a translation problem.
Similarly, single-author works have shown that this discretization enforces structural regularities that continuous models struggle to maintain. Karl D. D. Willis demonstrated in Engineering Sketch Generation for Computer-Aided Design that mapping geometric coordinates to a discrete grid forces generative models to naturally capture technical design constraints—such as perfectly horizontal or vertical alignments.
Building on this, Siyu Wang introduced CAD-GPT, which quantizes 3D space and rotation angles into a specialized geometric alphabet. This allows an MLLM to output valid, step-by-step CAD construction sequences, bypassing the precision limitations and floating-point drift that cause general-purpose continuous models to fail.
Conclusion
Discrete spatial tokenization works because it respects the architectural strengths of the Transformer. Rather than forcing a sequence model to perform continuous regression, it reformulates space into the very language the Transformer was born to speak: a structured sequence of discrete tokens, optimized by cross-entropy, and bound by context.
Key Foundational Work
Classifier Labels as Language Grounding for Explanations (2018) — contributed 2 concept elements to this direction.