Trokens: Semantic-Aware Relational Trajectory Tokens for Few-Shot Action Recognition

Pulkit Kumar, Shuaiyi Huang, Matthew Walmer, Sai Saketh Rambhatla, Abhinav Shrivastava

Abstract

Few-shot action recognition remains a significant challenge in computer vision, primarily due to the difficulty of learning complex motion patterns from limited data. Current methods often over-rely on appearance features or utilize inefficient point-tracking strategies. We present Trokens (Trajectory Tokens), a framework that redefines motion modeling by employing semantic-aware point sampling and explicitly capturing relational trajectory geometry. By focusing on action-critical objects and modeling both intra- and inter-trajectory relationships, Trokens achieves state-of-the-art performance across six major benchmarks, including a 3.8% improvement on Something-Something-V2, while being substantially more computationally efficient.

Introduction

Identifying actions in video sequences with only one or five examples (few-shot learning) requires a model to distinguish fine-grained motion cues. Traditional approaches often struggle because they prioritize global appearance over localized motion. Recent shifts toward point tracking—following specific pixels across frames—offered a potential solution, but faced two critical hurdles: inefficient sampling and weak motion modeling.

Comparison of grid sampling versus semantic-aware sampling and the types of motion modeled by Trokens.

As illustrated in Figure 3, previous methods typically use a uniform grid to select points for tracking. This approach often wastes resources on static backgrounds while missing small, vital objects like tools or hands. Furthermore, even when points are tracked, models frequently fail to capture the rich geometric relationships between moving entities, treating trajectories merely as anchors for appearance features rather than primary sources of information.

Methods

Trokens addresses these limitations through a two-pronged approach that transforms raw trajectories into "Semantic-Aware Relational Trajectory Tokens."

Semantic-Aware Sampling

Instead of a rigid grid, Trokens utilizes features from a pre-trained DINOv2 model to cluster the video into semantically meaningful regions. This allows the model to adaptively distribute tracking points, ensuring a higher density of points on small, distinct objects that are likely central to the action, while sparse points are assigned to uniform background areas.

Relational Motion Modeling

Once points are tracked (via Co-tracker), the system extracts two distinct types of motion features:

  1. Intra-trajectory (HoD): A modified Histogram of Oriented Displacements describes the magnitude and direction of individual point movements over time.
  2. Inter-trajectory: The model calculates pairwise relative distances between all tracked points, capturing coordinated movements (e.g., a hand approaching a cup).
The Trokens architecture, including appearance token extraction, semantic-aware point sampling, relational trajectory token generation, and the Decoupled Space-Time Transformer.

Decoupled Space-Time Transformer

As shown in the architecture diagram (Figure 4), these explicit motion features are fused with appearance tokens and processed by a Decoupled Space-Time Transformer. This architecture separately handles temporal and spatial relationships, providing a strong inductive bias that simplifies the learning task in data-scarce few-shot scenarios.

Key Results and Discussion

Trokens establishes new state-of-the-art (SOTA) benchmarks across six diverse datasets. On the motion-heavy Something-Something-V2 (SSV2) Full split, it outperformed the previous best method (TATs) by 3.8% in 1-shot settings. Even more striking was the 9.8% jump in 1-shot accuracy on the HMDB51 dataset.

Beyond accuracy, Trokens is highly efficient. The authors discovered that tracking just 32 semantically-chosen points outperformed tracking 256 points on a uniform grid. This efficiency translates to an 82% reduction in inference-time FLOPs compared to high-density uniform sampling methods. Qualitatively, the model successfully ignores static environments to focus on "action-relevant" objects, such as a lemon being sliced or a bottle being moved.

Conclusion

Trokens demonstrates that "smart sampling" is a force multiplier for video understanding. By providing explicit motion descriptors as an inductive bias, the framework enables Transformers to learn complex actions from minimal data. Future work will explore 3D point tracking to improve robustness against camera ego-motion and rapid perspective changes.

Summary updated

BibTeX
@inproceedings{kumar2025trokens,
  author = {Pulkit Kumar and Shuaiyi Huang and Matthew Walmer and Sai Saketh Rambhatla and Abhinav Shrivastava},
  title = {{Trokens: Semantic-Aware Relational Trajectory Tokens for Few-Shot Action Recognition}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2025},
}
Figure