Deep learning models excel at video classification when provided with thousands of labeled examples. However, in real-world applications—such as interactive robotics (Melo et al.), clinical therapy (Sabater et al.), or sign language mapping (Joshi et al.)—collecting massive video datasets is prohibitively expensive or impossible.
Why is learning an action from a single or few examples (few-shot learning) so difficult in video compared to static images? Videos are inherently high-dimensional, containing complex temporal dependencies, background clutter, and significant kinematic variability (Jin et al.). When a model sees only one example of an action, it struggles to separate the invariant essence of the movement from the idiosyncratic noise of the individual performer, leading to severe overfitting (Anqi Zhu et al.; Yan et al.).
To bypass this data hunger, modern few-shot action recognition frameworks shift away from learning categories from scratch. Instead, they rely on two core principles: decomposing actions into reusable atomic primitives and anchoring sparse visual features with rich semantic language priors.
Decomposing Actions into Atomic Primitives
Instead of treating every action category as a monolithic, unique entity, a more powerful approach breaks actions down into fundamental building blocks. If a model already understands basic movements like lift, extend, or grasp, it can recognize a completely novel action simply by identifying a new sequence or combination of these familiar primitives.
This compositional approach drastically reduces the sample complexity of learning new categories. For instance, Zhou et al. proposed a multi-task learning framework that decomposes complex actions into shared "latent tasks"—basic motion patterns like jumping or arm-swinging. By sharing these latent patterns across related categories via sparsity regularizers, the model can learn complex motions from a small number of samples.
Taking this further, Dessalene et al. revived the concept of "Therbligs"—atomic motion primitives from industrial engineering (such as Grasp, Move, and Release). By enforcing explicit logical constraints and hierarchical structures over these primitives, models can generalize effectively in low-data regimes without needing to see thousands of full action sequences.
[Monolithic View] "Pouring Coffee" --> Requires 1000s of full video examples
[Compositional View] "Pouring Coffee" --> [Grasp Pot] + [Lift] + [Tilt] + [Release]
(Reuses primitives learned across other tasks)
This decomposition works equally well across different modalities:
- Skeleton Streams: In skeleton-based gesture recognition, Jianbo Liu et al. decoupled gestures into two independent streams: hand posture variations (static shapes) and hand movements (global trajectories). Similarly, Anqi Zhu et al. introduced the Adaptive Local-Component-aware Graph Convolutional Network (ALCA-GCN), which breaks down 3D skeleton actions into localized spatial-temporal components, using attention to prioritize action-critical movements over performer-specific noise.
- Instructional Verbs and Nouns: In instructional videos, Zhukov et al. demonstrated that breaking actions down into shared verb and noun components allows a model to generalize to entirely unseen procedures by recombining known elements.
Resolving Temporal Misalignment
Even if a model identifies the correct primitives, different performers execute actions at varying speeds, durations, and temporal orderings. In a few-shot setting, a model cannot rely on a large dataset to smooth out these variations statistically. It must explicitly align the temporal dynamics of the few available "support" videos with the "query" video.
To address this, Bishay et al. introduced the Temporal Attentive Relation Network (TARN), which utilizes a segment-by-segment attention mechanism to dynamically align videos of different lengths, learning a deep distance metric to compare them directly. Rather than relying on a single global video descriptor, Linchao Zhu et al. proposed the Compound Memory Network (CMN), which stores videos as multi-faceted matrices and uses an abstract-to-constituent memory structure to match query videos at multiple temporal scales.
When spatial and temporal variations compound, parallel attention mechanisms become necessary. Xingyu Liu et al. developed PAINet, which deploys two parallel attention branches: spatial cross-attention to isolate subtle joint movements and temporal cross-attention to align inconsistent motion patterns over time. By resolving these misalignments structurally, the network prevents the model from overfitting to the specific timing of the support samples.
Anchoring Visuals with Semantic Language Priors
One of the most powerful strategies for sample-efficient learning is to leverage cross-modal knowledge. Humans do not look at a video in a vacuum; we use the semantic meaning of the action's name to guide our visual attention. By bridging the gap between raw pixels and linguistic concepts, models can bootstrap their understanding of rare actions.
Yifei Chen et al. introduced attentive semantic units to map fine-grained linguistic concepts directly to video frames, providing a strong inductive bias that constrains how visual features are interpreted in few-shot scenarios. With the advent of large vision-language models (VLMs), language can act as a robust classifier backbone. Wenhao Wu et al. proposed a "Text-as-Classifier" approach, where fixed textual embeddings of class labels are used directly as the weights of the video classification head. This prevents the model from warping its feature space destructively during fine-tuning on sparse data.
To inject even deeper domain knowledge, Yan et al. introduced CrossGLG, a framework where Large Language Models (LLMs) generate detailed textual descriptions of how an action is physically performed (e.g., describing exactly which joints move during a "handshake"). These descriptions guide 3D skeleton-based models to focus on key body joints, allowing the system to grasp the underlying principle of a movement from a single visual demonstration.
[Visual Input] --> Sparse, noisy 3D skeleton joints (High variance)
|
v (Cross-attention alignment)
[LLM Guidance] --> "An action where two people extend their right hands,
clasp them, and move them up and down." (Invariant prior)
Hallucination and Feature Borrowing
When data is extremely scarce, another intuitive solution is to artificially enrich the support set by "borrowing" features from common classes or hallucinating missing dimensions.
For example, if a model only has access to a still image of an action, it lacks the temporal dimension entirely. Yali Wang et al. solved this via a Hybrid Video Memory (HVM) machine, which "hallucinates" missing temporal motion cues for a static image by retrieving motion patterns from a pre-existing memory bank of unrelated videos.
In long-tailed video distributions where a few classes have many samples and most have very few, Perrett et al. introduced Long-Tail Mixed Reconstruction (LMR). LMR explicitly borrows sub-visual features from well-represented "head" classes to synthetically reconstruct and enrich the representations of data-scarce "tail" classes, ensuring robust generalization across the entire distribution.
Finally, this capacity to generalize from minimal data allows models to operate in highly flexible environments. Rosasco demonstrates that one-shot skeleton-based models can be successfully extended to open-set scenarios, allowing a system to recognize a single demonstration of a novel action while safely rejecting unknown background movements or unrelated gestures.
Key Foundational Work
Human Activity Recognition with Metric Learning (2008) — contributed 1 concept elements to this direction.