Spatial Feature Grids for Deep Reinforcement Learning

This direction is no longer present in the current build. Its last version stays readable and citable here.

Version 1 of 159 papers62 conceptsfrozen 2026-10-04

In traditional reinforcement learning, we often represent the state of an environment as a "flat" vector—a simple list of numbers. While this works for low-dimensional problems, it fails to capture the inherent geometry of complex environments like battlefields, city grids, or silicon chips.

Recent research across diverse fields has converged on a more powerful alternative: representing the environment as a multi-channel spatial grid. By encoding semantic information into separate layers of a 2D or 3D tensor, researchers are enabling neural networks to "see" the structural relationships of a problem, allowing them to solve tasks that were previously computationally intractable.

The Stack of Transparencies Intuition

Imagine you are playing StarCraft II. A flat vector might tell you that you have 10 units at coordinate $(x,y)$ and the enemy has 5 units at $(z,w)$. But this forces the neural network to manually reconstruct the concept of "distance" and "proximity" from raw coordinates.

As Oriol Vinyals and colleagues demonstrated in StarCraft II: A New Challenge for Reinforcement Learning, a better approach is to provide the agent with a stack of 2D feature grids. Think of these as a stack of overhead transparencies:

  • Layer 1: A binary map of terrain (walkable vs. blocked).
  • Layer 2: A heatmap of unit health.
  • Layer 3: A mask showing player ownership (friend vs. foe).

This structured representation allows Convolutional Neural Networks (CNNs) to apply the same "filters" across the entire map. If a network learns that a cluster of low-health enemies is a good target for an area-of-effect spell, it can apply that intuition anywhere on the map without needing to relearn it for every possible coordinate.

Why Spatial Encoding Works

The power of this approach lies in three core properties of convolutional architectures when applied to spatial grids:

  1. Translation Invariance: A tactical maneuver (like "kiting" in StarCraft) is the same whether it happens in the top-left or bottom-right of the screen. Yue Hu showed in Knowledge-Guided Agent-Tactic-Aware Learning for StarCraft Micromanagement that these maps allow agents to mirror human-like spatial reasoning.
  2. Preservation of Topology: In pathfinding, knowing that two points are adjacent is critical. Devon Sigurdson and Sven Koenig demonstrated in Automatic Algorithm Selection In Multi-agent Pathfinding that CNNs can "look" at a map and instantly recognize congestion patterns that a vector-based model would miss.
  3. Local Heterogeneity: In biological simulations, global averages (like "average chemical concentration") hide the local gradients that drive cell behavior. A. Bertin and the PhysiGym team found that representing chemical gradients as spatial images significantly improves the performance of agents learning to treat diseases like cancer.

Generalizing Beyond Games

While games like Snake (Ian Gemp in AlphaSnake) and Mahjong (Fuminori Okuya in Building a Computer Mahjong Player via Deep Convolutional Neural Networks) are natural fits for grid representations, the technique is being applied to physical engineering.

In semiconductor design, Yao Lai and the ChiPFormer team represent the silicon canvas as multi-channel masks. These masks track occupied space and estimated wirelength impacts, turning chip placement into a visual optimization problem. Similarly, Ke Xue used this "visual encoding" in MaskRegulate to refine existing layouts, treating the chip's physical constraints as layers in a spatial tensor.

The Complexity of Spatial Logic

The shift toward spatial representations isn't just a matter of convenience; it addresses a fundamental computational hardness. Christoph Dürr and Marek Chrobák proved in Tile-Packing Tomography Is NP-hard that reconstructing shapes from projections is NP-complete. By using multi-channel grids, researchers are essentially providing neural networks with the "modular blocks" needed to navigate these NP-hard search spaces more efficiently.

Whether it's routing droplets on a biochip (Tung-Che Liang in Adaptive Droplet Routing in Digital Microfluidic Biochips) or managing traffic lights (Hua Wei in A Survey on Traffic Signal Control Methods), the conclusion is the same: when a problem has a physical or logical layout, don't flatten it. Let the model see the space.

Papers in this version

History & citationVersion 1 · 2026-10-04

No longer present in the current build · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_spatial_feature_grids_for_deep_reinforce_5eafd9ea,
  title = {Spatial Feature Grids for Deep Reinforcement Learning},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/spatial-feature-grids-for-deep-reinforcement-learning/rdr_5eafd9ea38854bd281b312938ff80cd7},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_spatial_feature_grids_for_deep_reinforce_5eafd9ea_v1,
  title = {Spatial Feature Grids for Deep Reinforcement Learning},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/spatial-feature-grids-for-deep-reinforcement-learning/rdr_5eafd9ea38854bd281b312938ff80cd7/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}