Vectorized Spatial Representations for Geometric Reasoning

31 concepts21 papers2 proposals

When humans look at a floorplan, a city block, or a highway map, we do not perceive them merely as grids of colored pixels. Instead, we instinctively understand them as collections of discrete, interacting geometric entities: walls that meet at crisp right angles, land parcels bounded by clear property lines, and lanes that connect continuously across intersections.

Yet, for years, standard deep learning approaches treated spatial layout tasks through the lens of computer vision, processing environments as rasterized images or dense occupancy grids. While effective for pixel-level classification, this raster-based paradigm introduces severe limitations when applied to structured, geometric environments. A single pixel gap in a generated image can break a wall's containment, disconnect a traffic lane, or blur a precise architectural boundary. Furthermore, rasterized grids waste immense computational resources by allocating memory and processing power to empty space.

To overcome these challenges, a powerful paradigm shift has emerged: vectorized spatial representation. By encoding environments directly as structured geometric primitives—such as points, lines, polylines, polygons, and graphs—models can preserve exact topological relationships, enforce structural integrity, and drastically reduce computational overhead.


The Problem with Pixels: Topological Fragility

To understand why vectorized representations work, we must first examine why pixel-based approaches fail in structured domains. As reviewed by Liu et al. in their survey of computer-aided layout generation, standard image synthesis techniques frequently fail to preserve the topological integrity and precise geometric boundaries required for functional designs (Computer-Aided Layout Generation for Building Design: A Review).

Consider the task of generating a residential floorplan. A pixel-based generative model (like a standard GAN or pixel-diffusion network) outputs continuous pixel intensities. It possesses no inherent concept of a "wall" or a "room closure." Consequently, it often produces jagged boundaries, fuzzy corners, or hallucinated structural elements.

Raster Representation (Pixel Grid)        Vector Representation (Polygonal Loops)
+-----------------------------------+     +-----------------------------------+
| ████████████████      ███████████ |     | p1 ──────────────────────────── p2|
| ██            ██      ██       ██ |     | │                               │ |
| ██   Room A   ██      ██ Room B██ |     | │            Room A             │ |
| ████████████████      ███████████ |     | p4 ──────────────────────────── p3|
+-----------------------------------+     +-----------------------------------+
(Fragile: Small pixel gaps break topology) (Robust: Explicit coordinates & connectivity)

Vectorized architectures bypass this entirely by changing the output space from continuous pixels to discrete geometric entities. For example, Shabani et al. developed a framework that represents floorplans as sets of 1D polygonal loops (HouseDiffusion). Because the model directly generates the vertices of these loops, the output strictly adheres to the requested number of rooms and guarantees closed, non-rectilinear geometries without any rasterization artifacts.

Similarly, Liu et al. utilize a graph-constrained framework paired with a differentiable geometric loss to generate floorplans directly from room connectivity graphs (End-to-End Graph-Constrained Vectorized Floorplan Generation with Panoptic Refinement). By operating in a vector space, the network enforces architectural constraints—such as perfectly straight walls and aligned room volumes—that are incredibly difficult to guarantee via pixel manipulation.

This structural preservation extends to larger scales, such as urban planning. When synthesizing city blocks, rasterized approaches often suffer from fuzzy boundaries that obliterate small land parcels. To solve this, Xu et al. proposed representing land lots as nodes within a vectorized "ring topology" graph (BlockPlanner). This mathematical formulation ensures that even the smallest land parcels maintain geometric precision and perfect boundary alignment with their neighbors.


Computational Efficiency: Eliminating Empty Space

Beyond structural fidelity, vectorized representations offer a massive computational advantage by exploiting the inherent sparsity of spatial layouts.

In autonomous driving, high-definition (HD) maps cover vast geographic areas, but the critical features—lanes, crosswalks, and boundaries—occupy only a fraction of that space. Traditional systems rasterized these maps into high-resolution images to feed them into Convolutional Neural Networks (CNNs). This meant allocating significant compute to process empty asphalt, fields, and sky.

Rasterized HD Map (Dense Grid)             Vectorized HD Map (Sparse Polylines)
+-----------------------------------+     +-----------------------------------+
| . . . . . . . . . . . . . . . . . |     |        Polyline 1 (Lane Boundary) |
| ───────────────────────────────── |     |  p1 ─────────── p2 ─────────── p3 |
| . . . . . . . . . . . . . . . . . |     |                                   |
| ───────────────────────────────── |     |  q1 ─────────── q2 ─────────── q3 |
| . . . . . . . . . . . . . . . . . |     |        Polyline 2 (Lane Center)   |
+-----------------------------------+     +-----------------------------------+
(Compute wasted on empty space '.')       (Compute focused entirely on features)

Gao et al. fundamentally challenged this practice by introducing VectorNet (VectorNet: Encoding HD Maps and Agent Dynamics From Vectorized Representation). Instead of rasterizing the environment, VectorNet parameterizes road geometry and agent trajectories as discrete polylines. A hierarchical graph neural network then processes these vectors directly. By eliminating the need to process empty spatial regions, the authors reduced model parameters by 70% and cut computational operations (FLOPs) by an order of magnitude while improving prediction accuracy.

This insight has transformed motion forecasting and scene understanding:

  • Topology-Aware Routing: Liang et al. demonstrated that encoding HD maps as vectorized "lane graphs" allows models to explicitly incorporate road topology, such as lane connectivity, into the prediction process (Learning Lane Graph Representations for Motion Forecasting). This avoids the information loss and blurring inherent in pixel-based feature extraction, allowing for more accurate localization of actors relative to lane segments.
  • Long-Range Fidelity: As surveyed by Liu et al., polyline-based encodings are uniquely capable of preserving long-range road geometry and topological connectivity over extended horizons, a feat that becomes prohibitively expensive with high-resolution raster grids (A Survey on Deep-Learning Approaches for Vehicle Trajectory Prediction in Autonomous Driving).
  • End-to-End Planning Speed: For real-time autonomous driving, Jiang et al. developed VAD, an end-to-end vector-based planning model (VAD: Vectorized Scene Representation for Efficient Autonomous Driving). By replacing dense rasterized grids with structured coordinates for agents and map elements, VAD preserves distinct instance geometries and runs up to 9.3x faster than previous leading grid-based methods.

How It Works: Sequences, Graphs, and Primitives

How do neural networks natively process and generate vector data? The field relies on two primary mechanisms: sequential tokenization and primitive-level attention.

Spatial Layouts as Language

Because vector graphics are ultimately ordered sequences of coordinates and commands (like SVG paths), they can be modeled using sequence-to-sequence architectures originally designed for natural language processing.

In this formulation, a floorplan is treated as a string of text, where each "word" is a line segment coordinate. Ludvig Ericson applied this concept to robotics mapping with FloorGenT, demonstrating that modeling floor plans as sequences of line segments allows Transformers to capture long-range architectural dependencies, such as parallel or collinear walls, keeping structural boundaries perfectly sharp (FloorGenT: Generative Vector Graphic Model of Floor Plans for Robotics).

Ludvig Ericson expanded this to spatial prediction in Floorist, a model that allows robots to "hallucinate" unobserved parts of a building layout (Beyond the Frontier: Predicting Unseen Walls From Occupancy Grids by Learning From Floor Plans). By predicting a sequence of vector line segments rather than regressing a rasterized occupancy grid, the robot generates cleaner, structurally sound environmental models with significantly higher computational efficiency.

Primitive-Level Attention

Instead of forcing a model to look at arbitrary square patches of an image (as in standard Vision Transformers), vectorized architectures allow attention mechanisms to operate directly on meaningful geometric primitives.

Vision Transformer (ViT) Attention         Vector Primitive Attention
+---+---+---+---+                          +───────────────────────────────────+
|   |   |   |   |  (Attends to             |      / \                          |
+---+---+---+---+   arbitrary              |     /   \   (Attends directly     |
|   | █ |   |   |   square patches)        |    /_____\   to geometric lines,  |
+---+---+---+---+                          |   [_______]  arcs, and symbols)   |
|   |   |   |   |                          |                                   |
+---+---+---+---+                          +───────────────────────────────────+

For technical document analysis, Fan et al. introduced CADTransformer, which treats the native lines and arcs of a CAD drawing as discrete tokens (CADTransformer: Panoptic Symbol Spotting Transformer for CAD Drawings). By employing a neighborhood-aware attention mechanism over these vector primitives, the model reasons about the exact mathematical relationships between strokes, bypassing the limitations of pixel-based bounding boxes and significantly improving symbol identification in complex technical drawings.

Vectorization also enables seamless cross-modal interaction. Zeng et al. developed a point-and-vector spatial representation within their CARD framework (CARD: Cross-modal Agent Framework for Generative and Editable Residential Design). Because the layout is stored as discrete coordinates rather than a pixel grid, text-based LLM agents can communicate directly with visual generative models to perform highly precise, localized coordinate editing based on natural language feedback.

This mathematical focus is even effective in fine-grained design domains like typography. Wasim et al. demonstrated that representing fonts as vector paths rather than pixel grids allows machine learning models to capture the underlying mathematical construction of character strokes, avoiding raster noise and enabling highly accurate font similarity clustering (Toward Automatic Typography Analysis: Serif Classification and Font Similarities).


Downstream Interoperability and Real-World Metrics

The final, pragmatic reason vectorized representations work so well is their immediate compatibility with engineering pipelines. In professional workflows, a raster mask is an informational dead end; engineers and architects cannot easily import a JPEG into CAD or GIS software to calculate real-world metrics or build 3D models.

To address this, Kalervo et al. introduced the CubiCasa5K dataset, providing 5,000 annotated floorplans directly in structured SVG vector formats (CubiCasa5K: A Dataset and an Improved Multi-Task Model for Floorplan Image Analysis). This dataset shifted the focus of document analysis away from mere pixel classification and toward producing usable geometric data for CAD and 3D modeling.

When working with legacy raster archives, specialized pipelines are required to bridge the gap. Kippers et al. designed an automated system that pairs semantic segmentation with morphological post-processing to convert pixel blobs from old floorplans into clean, intersecting vector polygons, ensuring direct compatibility with 3D GIS environments like CityJSON (AUTOMATIC 3D BUILDING MODEL GENERATION USING DEEP LEARNING METHODS BASED ON CITYJSON AND 2D FLOOR PLANS).

Furthermore, storing layouts as vector coordinates allows models to calculate real-world physical metrics natively during training and evaluation. As demonstrated by Chen et al. with the ReCo dataset for urban planning, vector-based formats preserve exact coordinates and building heights (ReCo: A Dataset for Residential Community Layout Planning). This allows generative models to directly compute and optimize critical real-world constraints—such as building coverage ratios, fire safety distances, and solar access—metrics that are highly error-prone when estimated by counting pixels in an image.

By treating space not as an arbitrary grid of colors, but as a structured language of geometry, vectorized representations allow deep learning models to reason about the world with the same precision, efficiency, and structural awareness as the humans who designed it.

Key Foundational Work

CubiCasa5K: A Dataset and an Improved Multi-Task Model for Floorplan Image Analysis (2019) — contributed 1 concept elements to this direction.
Depicting stylized materials with vector shade trees (2013) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_vectorized_spatial_representation_a869ab7e,
  title = {vectorized spatial representation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/vectorized-spatial-representation/rdr_a869ab7e4e234ed9ba80a0119fa0a406},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_vectorized_spatial_representation_a869ab7e_v1,
  title = {vectorized spatial representation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/vectorized-spatial-representation/rdr_a869ab7e4e234ed9ba80a0119fa0a406/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake