Autoregressive Generation of Vector Geometry

25 concepts18 papers1 proposal

When humans draw a map, sketch a CAD model, or trace a shape, we do not fill in a grid of pixels one by one. Instead, we think in terms of structure: we place a vertex, draw a line segment, curve a stroke, and close a loop.

Traditionally, computer vision models have ignored this intuition, opting instead for rasterized outputs like pixel-wise segmentation masks or dense voxel grids. While easy to train via standard convolutional architectures, raster outputs suffer from fundamental flaws: they exhibit "jittery" or blurry boundaries, scale poorly with memory, and lack explicit topological awareness. To turn these pixel grids into the clean vector formats required by downstream applications like autonomous driving planners or engineering software, developers must rely on complex, hand-crafted post-processing pipelines.

A rapidly growing body of research circumvents this entirely by framing geometry generation as an autoregressive sequence modeling task. By serializing geometric structures into sequences of discrete tokens—such as coordinate pairs, drawing commands, or indexing pointers—models can directly "write" vector geometry.

How Geometry Becomes a Language

To apply sequence models like Recurrent Neural Networks (RNNs) or Transformers to spatial problems, continuous geometric data must first be translated into a discrete vocabulary.

As demonstrated by Ting Chen et al. in A Unified Sequence Interface for Vision Tasks, continuous spatial coordinates can be quantized into discrete bins, effectively turning coordinates into "words." Under this formulation, an instance segmentation mask is no longer a dense binary grid; it is an ordered sequence of vertex tokens that trace the object's boundary.

Continuous Space       Quantized Grid       Serialized Sequence
  (0.23, 0.87)     -->    [Bin 23, Bin 87]  -->   Token_23, Token_87, ...

Once geometry is serialized, a model can generate shapes step-by-step. At each time step, the network predicts the next token conditioned on the entire history of tokens generated so far. This autoregressive formulation allows the network to internalize the underlying "geometric grammar" of the target domain.

Capturing Structural Grammar and Continuity

Why does sequential generation work so well for geometric structures? The answer lies in the power of conditional context. In a traditional segmentation model, the decision to label a pixel as a boundary is largely independent of a pixel ten steps away. In an autoregressive model, the next point is explicitly constrained by the path that preceded it.

This sequential dependency ensures high continuity and topological consistency across a variety of domains:

Scaling Beyond Simple Polylines

The sequence vocabulary does not need to be limited to simple $(x, y)$ coordinates. By expanding the token vocabulary to include drawing commands, primitive parameters, or indexing pointers, this paradigm scales to highly complex geometric tasks.

Commands and Primitives

In computer-aided design (CAD) and typography, shapes are defined by rich vector formats like SVGs. Karl D. D. Willis et al. developed frameworks like CurveGen and TurtleGen in Engineering Sketch Generation for Computer-Aided Design that express CAD sketches as a sequence of pen-based commands (such as lines and arcs). Similarly, for typography, Haruka Aoki et al. (SVG Vector Font Generation for Chinese Characters with Transformer) leverage sequence-to-sequence models to emit the variable-length paths and commands required to render intricate Chinese characters cleanly at any scale.

Variable-Length Adaptability

Because these models generate tokens until a specific "end-of-sequence" token is reached, they adapt dynamically to the complexity of the target object. For instance, in scene text detection, Xiaobing Wang (Arbitrary Shape Scene Text Detection With Adaptive Text Region Representation.) uses an RNN to trace text boundaries by emitting a variable number of point pairs, effortlessly adjusting to long, curved, or irregular text lines. In autonomous driving, Yicheng Liu et al. (VectorMapNet: End-to-end Vectorized HD Map Learning) and Justin Liang (Convolutional Recurrent Network for Road Boundary Extraction) apply this to High-Definition (HD) map generation, predicting polyline vertices one by one to capture complex road geometries directly in a format self-driving planners can execute.

Extension to 3D Space

The same autoregressive logic applies to three dimensions. Zhizhong Han et al. (SeqXY2SeqZ: Structure Learning for 3D Shapes by Sequentially Predicting 1D Occupancy Segments from 2D Coordinates) represent 3D shapes by predicting sequences of 1D occupancy segments along spatial tubes, bypassing the cubic memory costs of 3D voxel grids.

For full 3D mesh reconstruction, Yujia Liu et al. (Point2Building: Reconstructing Buildings from Airborne LiDAR Point Clouds) decompose the problem into two sequential tasks: first generating a sequence of discrete vertex coordinates, and then predicting sequences of vertex indices to form polygonal faces.

Even physical properties can be composed sequentially; Zhijian Liu et al. (Physical Primitive Decomposition) employ an RNN to sequentially predict the size, position, rotation, and density of simple geometric building blocks to represent complex physical objects.

3D Mesh Generation Pipeline (Point2Building)
[Point Cloud] 
     │
     ▼
[Task 1: Predict Vertex Coordinates Sequence] ──► (x1,y1,z1), (x2,y2,z2), ...
     │
     ▼
[Task 2: Predict Face Index Pointers Sequence] ──► (v1,v2,v3), (v2,v3,v4), ...
     │
     ▼
[Final Vector Mesh]

The Intuition: The Model as a Dynamic Constraint Solver

Ultimately, autoregressive vector generation works because the model's internal attention mechanism or recurrent state acts as a dynamic constraint solver.

When a traditional segmentation network predicts a pixel, it lacks an explicit global guarantee of shape. In contrast, as Maxim Khomiakov et al. explain in GeoFormer: A Multi-Polygon Segmentation Transformer, modeling corners as spatially dependent tokens allows the network to reason about the geometry as a whole.

If a transformer has already generated three sides of a sharp rectangular building footprint, the conditional probability distribution for the next token becomes highly peaked around the exact coordinate required to close the rectangle cleanly. By learning the structural dependencies of shapes directly from data, autoregressive models replace rigid geometric templates and messy post-processing heuristics with a unified, elegant sequence interface.

Key Foundational Work

Annotating Object Instances with a Polygon-RNN (2017) — contributed 1 concept elements to this direction.
Physical Primitive Decomposition (2018) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_sequential_path_modeling_725205ef,
  title = {sequential path modeling},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/sequential-path-modeling/rdr_725205ef9a1b461d8ceb9080bad7830f},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_sequential_path_modeling_725205ef_v1,
  title = {sequential path modeling},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/sequential-path-modeling/rdr_725205ef9a1b461d8ceb9080bad7830f/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake