Trajectory Aware Feature Alignment for Video Understanding

This direction is merged into another direction: temporal feature misalignment. Its last version stays readable and citable here.

Version 1 of 1103 papers110 conceptsfrozen 2026-10-04

In computer vision, the transition from images to video is often treated as a simple matter of adding a temporal dimension. However, a fundamental conflict exists between how neural networks process data and how the physical world moves. Most standard architectures—whether they use 3D convolutions, recurrent units, or temporal attention—operate on a pixel-centric assumption: they assume that the semantic content at a specific spatial coordinate $(x, y)$ remains consistent across consecutive frames.

In reality, objects move. A pixel representing a person’s head in frame $t$ might represent the background in frame $t+1$. When models ignore this motion, they suffer from "feature contamination" and "boundary blur," where information from different semantic parts (or the background) is incorrectly fused together. A growing body of research is shifting toward Trajectory-Aware Alignment, a paradigm where models explicitly track and align features along the path of motion before attempting to reason about them.

The Failure of Fixed Coordinate Operations

The core problem with standard temporal operations is spatial misalignment. As Hong Chang and Xinqian Gu observe in their work on Appearance-Preserving 3D Convolution, standard 3D kernels often "blur" the appearance of subjects. Because the kernel processes fixed spatial coordinates across a temporal window, it might inadvertently mix features from a person's head in one frame with their shoulder in the next. This degrades the fine-grained details necessary for tasks like person re-identification.

This issue extends to Transformers. Standard temporal attention mechanisms compare features at the same pixel coordinates across time. Changgu Chen and Junwei Shu argue in Motion-Zero that this fixed-coordinate approach causes models to lose track of fast-moving subjects, leading to motion blur or the complete disappearance of objects in generated videos.

Aligning Before Aggregating

To solve this, researchers are developing "Alignment-Before-Aggregation" strategies. The intuition is simple: find where the object moved, shift the features to a common reference frame, and only then perform temporal fusion.

1. Optical Flow and Warping

One of the most direct ways to achieve this is by using optical flow to "warp" features. Cristian Sminchisescu and David Nilsson pioneered this with the STGRU, which uses flow to physically move hidden state information so it aligns with the current frame's visual input. More recently, Haiming Zhu and Shengfeng He introduced TokenWarping, which warps the "Query" tokens in a diffusion model's attention layer. By ensuring the query is looking at the correct moved location of an object, they eliminate the structural drift and flickering common in video-to-video translation.

2. Trajectory Attention

Instead of warping the entire feature map, some models redefine the attention mechanism itself. Zeqi Xiao and colleagues proposed Trajectory Attention, which performs attention along predicted pixel paths rather than fixed grids. This allows the model to maintain long-range consistency because the "memory" of an object follows its physical movement through space. Similarly, Rui Sun and Feng Wu developed the Trajectory Memory Retrieval Network, which aligns historical memory frames with the current query frame based on the object's natural trajectory before segmenting.

When motion is complex or flow is noisy, models can use flexible search windows. Kent Gauen and Stanley H. Chan introduced Shifted Non-Local Search, which performs a small, differentiable grid search around predicted offsets to correct motion inaccuracies. This is echoed in the work of Fuchen Long and Ting Yao, whose Stand-Alone Inter-Frame Attention (SIFA) uses deformable sampling to dynamically "track" and aggregate features from deforming objects, preventing the contamination of foreground features with background noise.

Why Trajectory Awareness Works

The success of these methods stems from their adherence to physical constraints. In Few-shot Action Recognition, Huabin Liu and Yuxi Li show that predicting spatial offsets allows a model to compare the actual action across two videos, even if the actors are in different parts of the frame. Without this alignment, the model's similarity metrics are confounded by irrelevant background differences.

By moving from a "static grid" view of video to a "dynamic trajectory" view, these researchers are enabling models to:

  • Maintain Identity: Objects don't "flicker" or change appearance because the model is consistently looking at the same semantic part of the object across time.
  • Resolve Ambiguity: As shown by Bugra Tekin and Pascal Fua, motion-compensated sequences allow models to use temporal cues to resolve depth ambiguities and occlusions that are impossible to solve from a single frame.
  • Enable Control: In generative models like TrackDiffusion, trajectory-aware modules allow users to dictate exactly where an object should move, ensuring the generated pixels follow the intended path without losing their identity.

Ultimately, trajectory-aware alignment treats video not as a stack of images, but as a continuous flow of information, where the most important connections are found by following the motion, not just the pixels.

Papers in this version

Showing 60 of 103 papers.

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_trajectory_aware_feature_alignment_for_v_01b7d32c,
  title = {Trajectory Aware Feature Alignment for Video Understanding},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/trajectory-aware-feature-alignment-for-video-understanding/rdr_01b7d32cd621424fa25e74b51b00c007},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_trajectory_aware_feature_alignment_for_v_01b7d32c_v1,
  title = {Trajectory Aware Feature Alignment for Video Understanding},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/trajectory-aware-feature-alignment-for-video-understanding/rdr_01b7d32cd621424fa25e74b51b00c007/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}