Preserving Spatial Hierarchies with Capsule Networks

This direction is split into other directions: spatial information loss (spatial relationship preservation). Its last version stays readable and citable here.

Version 1 of 162 papers63 conceptsfrozen 2026-10-04

Standard convolutional neural networks (CNNs) are remarkably good at identifying what is in an image, but they are surprisingly bad at understanding where things are in relation to each other. This limitation stems from a fundamental design choice: the use of pooling layers to achieve translation invariance.

While pooling allows a network to recognize a feature regardless of its exact position, it does so by discarding precise spatial information. As noted in A Survey of Convolutional Neural Networks: Analysis, Applications, and Prospects, this "spatial blindness" prevents models from distinguishing between a correctly structured object and a "Picasso-like" jumble where the parts are present but misplaced.

The Problem with Invariance

In a traditional CNN, a max-pooling layer looks at a small window of neurons and passes only the strongest activation to the next layer. This tells the network that a feature (like an eye) exists somewhere in that window, but it throws away the eye's exact coordinates and orientation.

This leads to a vulnerability: models can be triggered by "Sparse Fooling Images"—nonsense patterns that contain the right features in the wrong places. Soichiro Kumano and colleagues demonstrated that DNNs can be fooled with 99% confidence by images that humans find unrecognizable, precisely because the models prioritize feature presence over global spatial context.

Capsules as Coordinate Frames

To solve this, researchers led by Geoffrey E Hinton and Sara Sabour proposed Capsule Networks (CapsNets). Instead of using a single scalar value to represent a feature's probability, a "capsule" uses a vector or a matrix to represent the feature's pose.

A capsule's output encodes two things:

  1. Existence: The length (magnitude) of the vector represents the probability that the entity exists.
  2. Instantiation Parameters: The orientation or internal values of the vector (or matrix) represent properties like position, size, and rotation.

This shift moves the network from invariance (ignoring changes) to equivariance (representing changes). If an object rotates in the input, the capsule's vector rotates accordingly, preserving the geometric information rather than washing it away.

Routing by Agreement

The most critical mechanism in this research direction is how capsules at one level "vote" for capsules at the next level. This is known as dynamic routing or routing-by-agreement.

Imagine a "nose" capsule and an "eye" capsule. Based on their own poses, they can each predict where the "face" they belong to should be. If their predictions for the face's pose align, they "agree" that a face exists. In Matrix capsules with EM routing, this is formalized using an Expectation-Maximization (EM) procedure where parts effectively cluster their votes to find the most likely pose of the whole.

This part-whole hierarchy is much more robust than simple feature detection. As Nicola Garau and co-authors argue, this allows the network to explicitly represent the structural hierarchy of how discrete parts compose a unified whole, making the model's internal logic more interpretable.

Why Spatial Intelligence Matters

The ability to preserve spatial relationships is not just a theoretical improvement; it solves practical bottlenecks in several domains:

  • Data Efficiency: Because CapsNets understand geometric transformations, they don't need to see an object from every possible angle during training. Amelia Jiménez-Sánchez showed that CapsNets are significantly more robust than CNNs when training on the tiny, imbalanced datasets common in medical imaging.
  • Precise Segmentation: In tasks like lung segmentation, Rodney LaLonde demonstrated that "SegCaps" could achieve state-of-the-art results with 95% fewer parameters than a standard U-Net by maintaining the spatial integrity of anatomical structures.
  • Complex Signal Analysis: Beyond images, this logic applies to any data with structural relationships. Jaesung Bae applied routing-by-agreement to speech spectrograms, finding that it better captured the temporal and frequency-based "geometry" of voice commands.

Overcoming Computational Costs

The primary drawback of early Capsule Networks was the computational intensity of the iterative routing process. Recent work has focused on making these architectures deeper and faster.

Jathushan Rajasegaran introduced DeepCaps, which uses 3D convolutions to stabilize routing, allowing the networks to scale to more complex datasets like CIFAR-10. Meanwhile, hardware-centric approaches like PIM-CapsNet propose offloading the heavy routing math to specialized Processing-In-Memory hardware, achieving massive speedups.

By replacing the "blunt instrument" of pooling with the "surgical precision" of coordinate-aware routing, this research direction seeks to build neural networks that perceive the world not as a bag of features, but as a structured hierarchy of objects and parts.

Papers in this version

Showing 60 of 62 papers.

History & citationVersion 1 · 2026-10-04

Split into other directions · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_preserving_spatial_hierarchies_with_caps_fbd4d4b8,
  title = {Preserving Spatial Hierarchies with Capsule Networks},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/preserving-spatial-hierarchies-with-capsule-networks/rdr_fbd4d4b8cc6243178569cd8c75a62d31},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_preserving_spatial_hierarchies_with_caps_fbd4d4b8_v1,
  title = {Preserving Spatial Hierarchies with Capsule Networks},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/preserving-spatial-hierarchies-with-capsule-networks/rdr_fbd4d4b8cc6243178569cd8c75a62d31/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}