When a human looks at two videos—one of a chef rapidly dicing an onion and another of a child slowly slicing a banana—they instantly recognize both as instances of "cutting." To a computer vision model, however, these two sequences look completely different. The raw pixels, joint velocities, and temporal durations share almost no overlap.
This is the core challenge of intra-class variation: the reality that a single semantic action category can manifest in an infinite variety of physical executions. As computer vision transitioned from sterilized laboratory settings to real-world environments, as highlighted by Laptev et al., managing this variability became a central bottleneck. Individuals perform the same action with unique speeds, styles, body proportions, and cultural nuances (Kong et al.; Noroozi et al.).
To build models that generalize across different people, researchers have shifted away from rigid templates toward representations that capture the invariant essence of an action.
Why Average Templates Fail
A naive approach to action recognition might involve creating a "prototype" or average template for an action and measuring how close a new video is to that template. However, Yao et al. demonstrated that a single action category encompasses too wide a range of distinct body configurations for an average template to be meaningful. If one person waves with a high elbow and another waves with a low elbow, their mathematical average might resemble someone holding their arm static, destroying the motion signature entirely.
This problem is exacerbated in fine-grained scenarios, such as micro-gestures (Chen et al.) or infant movements (Hatamimajoumerd et al.), where simple similarity-based ranking against prototypes fails to achieve high precision (Li et al.). When distinct physical movements are grouped into a single broad activity class, the resulting compounding of diverse motion signatures makes standard distance metrics brittle (Kleanthous et al.).
Mapping Actions as Geometric Trajectories
Instead of treating an action as a static point or a linear sequence of coordinates, a more powerful intuition is to view an action as a continuous path on a high-dimensional surface, or manifold. Even if two actors move at different speeds or have different arm lengths, the geometric shape of their movement path remains topologically similar.
[Raw Skeleton Sequence] ---> [Map to Riemannian Manifold] ---> [Bézier Curve Smoothing] ---> [Invariant Action Path]
To exploit this, Szczapa et al. modeled skeleton sequences as trajectories on the Riemannian manifold of positive semi-definite matrices. By applying Bézier curve smoothing to denoise the trajectory and using a Global Alignment Kernel, they could align and compare the shapes of movements independently of the actor's physical proportions or execution speed.
Similarly, Wang et al. broke continuous actions into short "action-snippets" and projected them onto geometric structures called Activated Simplices. This bounds the action space tightly, allowing the model to remain robust to individual style variations and noisy skeletal data.
Extracting Invariant Temporal Features
Another way to defeat intra-class variation is to isolate features that change slowly over time. When a person performs an action, their raw joint coordinates change rapidly and noisily from frame to frame. However, the underlying semantic state—the fact that they are "jumping"—changes very slowly.
Shan et al. applied Slow Feature Analysis (SFA) to 3D skeleton data to exploit this phenomenon. By expanding raw joint coordinates into multi-order streams (capturing distances and angles) and training the network to prioritize slowly varying signals, the framework effectively filters out the high-frequency noise introduced by individual execution styles and speeds.
For scenarios where data is scarce, such as few-shot video classification, explicit temporal alignment can be too computationally expensive. To solve this, Toby Perrett introduced the Temporal-Relational CrossTransformer (Temporal-Relational CrossTransformers for Few-Shot Action Recognition), which matches temporally ordered sub-sequences (tuples of frames) between a query video and support videos. This allows the model to recognize an action even if it occurs at different speeds or start times.
Taking this a step further, Zhang et al. developed an implicit temporal alignment network (ITANet) that uses factorized self-attention to bake context directly into frame-level features, allowing videos to be compared via simple summation rather than rigid alignment algorithms.
Disentangling Style from Semantics
An observed human movement is always a combination of two distinct components:
- The Semantics: What action is being performed.
- The Style: Who is performing it, how fast they are moving, and the camera angle.
If a model can mathematically separate these two components, it can discard the style noise and focus entirely on the semantics.
+---> [Semantic-Related Features] ---> Match to Text Label
|
[Raw Skeleton Data] -+
|
+---> [Style/Identity Noise] --------> Discard
The authors of SA-DVAE achieved this by using Disentangled Variational Autoencoders. The network explicitly splits skeleton data into "semantic-related" and "irrelevant" latent spaces, ensuring that only the action-defining components are aligned with text descriptions. This prevents variations in actor body types or recording perspectives from corrupting the classification.
When dealing with fine-grained micro-actions where variations are incredibly subtle, Cui et al. approached person-independence through the lens of optimization. They combined feature-level alignment with a loss function based on Distributionally Robust Optimization (DRO). Instead of optimizing for the average performance across a dataset, DRO forces the model to perform well on the most challenging, non-standard execution styles, ensuring robust generalization to unseen individuals.
Decomposing Actions into Semantic Attributes
Finally, instead of mapping raw pixels directly to a monolithic label like "cooking," models can benefit from an intermediate layer of interpretation. Liu et al. proposed breaking actions down into high-level semantic action attributes (e.g., "arm motion," "torso twist").
Because raw kinematic data is often the most inconsistent part of an action signal (Melo et al.), decomposing the video into these discrete, interpretable body part movements (Wang et al.) or behavioral building blocks (Nonparametric discovery of activity patterns from video collections) provides a buffer against visual variance. If a model knows what individual body parts are supposed to do, it can tolerate a wide variety of global execution styles, making the final classification far more stable.
Key Foundational Work
Parameterized Modeling and Recognition of Activities (1998) — contributed 1 concept elements to this direction.
Recognition and Segmentation of 3-D Human Action Using HMM and Multi-class AdaBoost (2006) — contributed 1 concept elements to this direction.