When humans draw a map, sketch a CAD model, or trace a shape, we do not fill in a grid of pixels one by one. Instead, we think in terms of structure: we place a vertex, draw a line segment, curve a stroke, and close a loop.
Traditionally, computer vision models have ignored this intuition, opting instead for rasterized outputs like pixel-wise segmentation masks or dense voxel grids. While easy to train via standard convolutional architectures, raster outputs suffer from fundamental flaws: they exhibit "jittery" or blurry boundaries, scale poorly with memory, and lack explicit topological awareness. To turn these pixel grids into the clean vector formats required by downstream applications like autonomous driving planners or engineering software, developers must rely on complex, hand-crafted post-processing pipelines.
A rapidly growing body of research circumvents this entirely by framing geometry generation as an autoregressive sequence modeling task. By serializing geometric structures into sequences of discrete tokens—such as coordinate pairs, drawing commands, or indexing pointers—models can directly "write" vector geometry.
How Geometry Becomes a Language
To apply sequence models like Recurrent Neural Networks (RNNs) or Transformers to spatial problems, continuous geometric data must first be translated into a discrete vocabulary.
As demonstrated by Ting Chen et al. in A Unified Sequence Interface for Vision Tasks, continuous spatial coordinates can be quantized into discrete bins, effectively turning coordinates into "words." Under this formulation, an instance segmentation mask is no longer a dense binary grid; it is an ordered sequence of vertex tokens that trace the object's boundary.
Continuous Space Quantized Grid Serialized Sequence
(0.23, 0.87) --> [Bin 23, Bin 87] --> Token_23, Token_87, ...
Once geometry is serialized, a model can generate shapes step-by-step. At each time step, the network predicts the next token conditioned on the entire history of tokens generated so far. This autoregressive formulation allows the network to internalize the underlying "geometric grammar" of the target domain.
Capturing Structural Grammar and Continuity
Why does sequential generation work so well for geometric structures? The answer lies in the power of conditional context. In a traditional segmentation model, the decision to label a pixel as a boundary is largely independent of a pixel ten steps away. In an autoregressive model, the next point is explicitly constrained by the path that preceded it.
This sequential dependency ensures high continuity and topological consistency across a variety of domains:
- Boundary Tracing: In End-to-End Line Drawing Vectorization, Hanyuan Liu et al. show that conditioning each new curve segment on the preceding geometry mimics human tracing behavior, preventing the disjointed or broken lines typical of non-sequential vectorization methods.
- Architectural Layouts: Ludvig Ericson demonstrates in Beyond the Frontier: Predicting Unseen Walls From Occupancy Grids by Learning From Floor Plans that sequence modeling allows a robot to reason logically about structural dependencies. For example, the model can "hallucinate" a wall extending behind a detected doorway because it understands the architectural grammar of floor plans, avoiding the phantom obstacles common in purely image-based predictions.
- Polygon Delineation: For tasks like extracting building footprints from satellite imagery, the sequence must form a closed loop. Early work like Polygon-RNN by Lluis Castrejon et al. (Annotating Object Instances with a Polygon-RNN) and subsequent frameworks by Wufan Zhao et al. (BUILDING OUTLINE DELINEATION: FROM VERY HIGH RESOLUTION REMOTE SENSING IMAGERY TO POLYGONS WITH AN IMPROVED END-TO-END LEARNING FRAMEWORK) proved that predicting an ordered list of vertices naturally produces sharp corners and straight edges. As Mingming Zhang et al. note in BiSVP: Building Footprint Extraction via Bidirectional Serialized Vertex Prediction, this serialization allows the network to output clean, ordered polygons directly, bypassing raster-to-vector post-processing entirely.
Scaling Beyond Simple Polylines
The sequence vocabulary does not need to be limited to simple $(x, y)$ coordinates. By expanding the token vocabulary to include drawing commands, primitive parameters, or indexing pointers, this paradigm scales to highly complex geometric tasks.
Commands and Primitives
In computer-aided design (CAD) and typography, shapes are defined by rich vector formats like SVGs. Karl D. D. Willis et al. developed frameworks like CurveGen and TurtleGen in Engineering Sketch Generation for Computer-Aided Design that express CAD sketches as a sequence of pen-based commands (such as lines and arcs). Similarly, for typography, Haruka Aoki et al. (SVG Vector Font Generation for Chinese Characters with Transformer) leverage sequence-to-sequence models to emit the variable-length paths and commands required to render intricate Chinese characters cleanly at any scale.
Variable-Length Adaptability
Because these models generate tokens until a specific "end-of-sequence" token is reached, they adapt dynamically to the complexity of the target object. For instance, in scene text detection, Xiaobing Wang (Arbitrary Shape Scene Text Detection With Adaptive Text Region Representation.) uses an RNN to trace text boundaries by emitting a variable number of point pairs, effortlessly adjusting to long, curved, or irregular text lines. In autonomous driving, Yicheng Liu et al. (VectorMapNet: End-to-end Vectorized HD Map Learning) and Justin Liang (Convolutional Recurrent Network for Road Boundary Extraction) apply this to High-Definition (HD) map generation, predicting polyline vertices one by one to capture complex road geometries directly in a format self-driving planners can execute.
Extension to 3D Space
The same autoregressive logic applies to three dimensions. Zhizhong Han et al. (SeqXY2SeqZ: Structure Learning for 3D Shapes by Sequentially Predicting 1D Occupancy Segments from 2D Coordinates) represent 3D shapes by predicting sequences of 1D occupancy segments along spatial tubes, bypassing the cubic memory costs of 3D voxel grids.
For full 3D mesh reconstruction, Yujia Liu et al. (Point2Building: Reconstructing Buildings from Airborne LiDAR Point Clouds) decompose the problem into two sequential tasks: first generating a sequence of discrete vertex coordinates, and then predicting sequences of vertex indices to form polygonal faces.
Even physical properties can be composed sequentially; Zhijian Liu et al. (Physical Primitive Decomposition) employ an RNN to sequentially predict the size, position, rotation, and density of simple geometric building blocks to represent complex physical objects.
3D Mesh Generation Pipeline (Point2Building)
[Point Cloud]
│
▼
[Task 1: Predict Vertex Coordinates Sequence] ──► (x1,y1,z1), (x2,y2,z2), ...
│
▼
[Task 2: Predict Face Index Pointers Sequence] ──► (v1,v2,v3), (v2,v3,v4), ...
│
▼
[Final Vector Mesh]
The Intuition: The Model as a Dynamic Constraint Solver
Ultimately, autoregressive vector generation works because the model's internal attention mechanism or recurrent state acts as a dynamic constraint solver.
When a traditional segmentation network predicts a pixel, it lacks an explicit global guarantee of shape. In contrast, as Maxim Khomiakov et al. explain in GeoFormer: A Multi-Polygon Segmentation Transformer, modeling corners as spatially dependent tokens allows the network to reason about the geometry as a whole.
If a transformer has already generated three sides of a sharp rectangular building footprint, the conditional probability distribution for the next token becomes highly peaked around the exact coordinate required to close the rectangle cleanly. By learning the structural dependencies of shapes directly from data, autoregressive models replace rigid geometric templates and messy post-processing heuristics with a unified, elegant sequence interface.
Key Foundational Work
Annotating Object Instances with a Polygon-RNN (2017) — contributed 1 concept elements to this direction.
Physical Primitive Decomposition (2018) — contributed 1 concept elements to this direction.