Topological Joint Ordering for Skeleton Based Action Recognition

12 concepts10 papers1 proposal

When humans observe an action, we naturally perceive the movement of limbs and the geometric constraints of the skeleton. However, teaching a computer vision model to recognize these actions from 3D coordinate data presents a fundamental structural challenge.

A human skeleton is inherently a graph—a network of joints connected by bones. Yet, the most powerful and computationally optimized deep learning architectures, such as 2D Convolutional Neural Networks (CNNs), are designed for grids. To bridge this gap, researchers must transform 3D coordinate sequences into 1D vectors or 2D images. How we order the joints during this transformation dictates whether a model succeeds or fails.

The Core Dilemma: Graphs vs. Grids

The power of a convolutional kernel lies in its local receptive field. A standard $3 \times 3$ convolution matrix operates under the assumption of spatial locality: pixels that are close together in the grid are highly correlated in the real world.

When dealing with skeleton data, the naive approach is to flatten the joint coordinates into a tensor based on their default indices in a dataset (e.g., Joint 1, Joint 2, ..., Joint 25). As noted by Yan et al. and Li et al., this arbitrary serialization completely destroys the spatial topology of the human body.

If a dataset happens to list the left hand joint next to the right foot joint, an arbitrary flattening forces the convolutional kernel to look at them simultaneously. Conversely, joints that are intimately connected—like the elbow and the wrist—might end up separated by dozens of elements in the tensor. As a result, standard convolutional filters waste their capacity trying to learn relationships across an unnatural layout that does not align with physical anatomy, as argued by Li et al..

Why Topology Preservation Matters

For a neural network to efficiently learn motion mechanics, physically connected joints must remain spatially adjacent within the input tensor. Pham et al. demonstrate that preserving these local spatial characteristics allows convolutional filters to naturally capture coherent motion patterns, such as the synchronized swing of an arm or the bending of a knee.

Arbitrary Ordering (Fragmented Topology):
[Hand] -> [Foot] -> [Spine] -> [Elbow]  <-- Kernel sees noise

Topological Ordering (Preserved Topology):
[Shoulder] -> [Elbow] -> [Wrist] -> [Hand] <-- Kernel captures limb mechanics

When the input data structure mirrors the physical body, the network does not have to spend parameters reconstructing the basic layout of a human. Instead, the first few layers of a CNN can immediately begin aggregating local joint relationships, gradually building up to global body movements as the receptive field expands.

The Solution: Depth-First Tree Traversals

To map a branching skeletal graph onto a linear sequence or a 2D grid without losing structural information, researchers turned to classical graph theory: depth-first search (DFS) tree traversal.

Yang et al. introduced this concept via Tree Structure Skeleton Images (TSSI). By treating the skeleton as a tree rooted at the spine, a depth-first traversal visits a joint, moves down to its continuous extremity, and then backtracks. This mathematical path guarantees that physically connected body parts remain adjacent in the resulting sequence.

To ensure complete continuity, some implementations utilize a cyclic path. Baradel et al. show that arranging 3D joint coordinates along a continuous, cyclic topological path allows convolutional filters and attention mechanisms to more effectively learn the underlying mechanics of motion.

This strategy has been successfully deployed across several distinct architectural frameworks:

  • Spatio-Temporal Images: Caetano et al. use depth-first tree traversals to generate 2D images where one axis represents time and the other represents the topologically ordered joints. This allows a relatively shallow, standard CNN to capture complex multi-scale motion. In subsequent work, Caetano et al. combined this traversal with reference joints to preserve relative spatial configurations.
  • Feature Concatenation: Even outside of pure image generation, ordering matters. Chen et al. demonstrate that concatenating feature vectors from individual body parts in a strict anatomical order ensures that reinforcement learning agents retain critical structural relationships throughout action prediction.
  • Fine-Grained Domains: The principle scales down to highly intricate movements. For instance, Laines et al. applied DFS-ordered skeleton images to isolated sign language recognition, proving that preserving spatial continuity allows standard, highly optimized 2D convolutional kernels to read complex hand gestures accurately.

By treating the ordering of joints not as a data formatting afterthought, but as a core architectural prior, topological joint ordering allows standard grid-based neural networks to process graph-structured biological motion with remarkable efficiency.

Key Foundational Work

Human Activity Recognition with Pose-driven Attention to RGB (2018) — contributed 2 concept elements to this direction.
Action Recognition with Visual Attention on Skeleton Images (2018) — contributed 2 concept elements to this direction.
Part-Activated Deep Reinforcement Learning for Action Prediction (2018) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_topological_joint_ordering_ab51571f,
  title = {topological joint ordering},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/topological-joint-ordering/rdr_ab51571fbca1479b92e7479ce5108c3d},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-22},
}

This exact version (version 1, frozen text):

@misc{lacuna_topological_joint_ordering_ab51571f_v1,
  title = {topological joint ordering},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/topological-joint-ordering/rdr_ab51571fbca1479b92e7479ce5108c3d/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-22},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake