Topological Ordering for Skeletal Sequence Modeling

This direction is split into other directions: topological joint ordering. Its last version stays readable and citable here.

Version 1 of 116 papers25 conceptsfrozen 2026-10-04

When we represent the human body for a computer, we usually treat it as a set of 3D coordinates—a "skeleton" of joints. However, standard deep learning architectures like Convolutional Neural Networks (CNNs) and Recurrent Neural Networks (RNNs) were designed for grids (images) or linear sequences (text). If you simply feed a list of joint coordinates into these models using an arbitrary numerical order, you force the model to learn the fundamental rules of human anatomy from scratch before it can even begin to recognize an action.

A growing body of research suggests that the key to efficient skeletal modeling is topological ordering: reordering joint data so that its sequence or grid position reflects the physical connectivity of the human body.

The Problem with Arbitrary Indexing

In a typical dataset, joints are assigned indices based on the sensor's output—for example, the head might be index 1, the left hand index 15, and the left elbow index 20. When these are processed as a sequence, the hand and elbow (which are physically connected) are separated by 14 unrelated data points.

As Lin Li and Zhaoxiang Zhang point out in Skeleton-Based Relational Modeling for Action Recognition, mapping skeletons to generic grids often severs the natural spatial links between joints. This makes it difficult for a model to maintain "structural integrity." If a convolution kernel or a recurrent hidden state doesn't "see" the hand and elbow together, it struggles to understand the constraints of a limb's movement.

Tree Traversal and Kinematic Chains

To solve this, researchers like Jun Liu introduced the Spatio-Temporal LSTM (ST-LSTM). In Spatio-Temporal LSTM with Trust Gates for 3D Human Action Recognition, Liu utilizes a tree-structure traversal (specifically a depth-first search) to order the joints.

By following the kinematic chain—moving from the root (the hip) down through the spine to the extremities—the model processes data in a kinematically logical order. This ensures that the "memory" of the LSTM moves along the limbs, preventing the model from learning spurious dependencies between unrelated body parts, like a foot and a shoulder, simply because they happened to be adjacent in a list.

Similarly, Xiangbo Shu demonstrated in Spatiotemporal Co-attention Recurrent Neural Networks for Human-Skeleton Motion Prediction that following the natural kinematic chain allows recurrent architectures to better capture the spatial constraints inherent in human movement, leading to more stable long-term motion predictions.

Turning Skeletons into Images

The same principle applies when using CNNs. To use powerful image-recognition backbones, we must transform 1D joint lists into 2D "skeleton images." Christian Wolf and Fabien Baradel showed in Pose-conditioned Spatio-Temporal Attention for Human Action Recognition that arranging joints topologically allows standard 2D convolutions to learn hierarchical spatial features.

When joints are ordered such that neighbors in the image are neighbors in the body, a $3 \times 3$ convolution kernel actually "sees" a local anatomical neighborhood. This idea was further refined by David Laines for Isolated Sign Language Recognition based on Tree Structure Skeleton Images. By using a depth-first search to populate the image grid, they ensured that physically connected joints remain adjacent, allowing local kernels to capture meaningful anatomical correlations essential for understanding complex hand gestures.

Geometry-Aware Scanning in State Space Models

The most recent evolution of this concept appears in State Space Models (SSMs). In PoseMamba: Monocular 3D Human Pose Estimation with Bidirectional Global-Local Spatio-Temporal State Space Model, Yunlong Huang and Junshuo Liu adapt the Mamba architecture for pose estimation.

Mamba relies on scanning sequences linearly. To make this work for the human body, they use topology-constrained scanning. Instead of a generic scan, the model "walks" through the skeletal branches. This geometry-aware strategy allows the model to capture both global posture and intricate local limb movements with linear complexity, outperforming Transformers which often struggle to prioritize these local structural constraints without massive amounts of data.

Why It Works: Inductive Bias

The success of topological ordering across LSTMs, CNNs, and SSMs stems from a shared principle: inductive bias. By structuring the input to match the physical reality of the subject, we reduce the search space for the neural network.

  1. Local Correlation: Most human movement is local (an elbow moves relative to a shoulder). Topological ordering places these correlated signals close together in the data stream.
  2. Hierarchical Learning: Physical movement is hierarchical (the hand's position is a function of the arm's position). Tree-based ordering allows models to build representations of limbs before building a representation of the whole body.
  3. Noise Reduction: By respecting the skeleton's graph topology, models are less likely to "hallucinate" relationships between joints that have no physical connection.

Papers in this version

History & citationVersion 1 · 2026-10-04

Split into other directions · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_topological_ordering_for_skeletal_sequen_5a8fdba7,
  title = {Topological Ordering for Skeletal Sequence Modeling},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/topological-ordering-for-skeletal-sequence-modeling/rdr_5a8fdba7acf44db7bdc705c329f17a31},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-22},
}

This exact version (version 1, frozen text):

@misc{lacuna_topological_ordering_for_skeletal_sequen_5a8fdba7_v1,
  title = {Topological Ordering for Skeletal Sequence Modeling},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/topological-ordering-for-skeletal-sequence-modeling/rdr_5a8fdba7acf44db7bdc705c329f17a31/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-22},
}