This paper addresses a fundamental flaw in how modern convolutional neural networks (CNNs) process video: the assumption that objects stay in the same place from one frame to the next. By introducing a novel operation that aggregates video features along motion paths rather than at fixed pixel locations, the proposed model achieves significant performance gains on action recognition tasks.
The Problem: Spatial Misalignment Across Time
In video analysis, the temporal dimension is typically handled using either computationally expensive 3D convolutions or "factorized" convolutions (like R(2+1)D or S3D). Factorized models split the process into a 2D spatial convolution followed by a 1D temporal convolution.
However, these factorized models rely on a major implicit assumption: they assume feature maps are perfectly aligned across time. If a person is at coordinates $(x, y)$ in frame 1, the network assumes the most relevant information for that person in frame 2 is also at $(x, y)$.
In reality, objects move, deform, and shift. A standard temporal convolution at a fixed location might aggregate a person's hand in frame 1 with the background in frame 2. This spatial misalignment makes it difficult for the network to learn the true dynamics of an action.
The Solution: Trajectory Convolution
The core innovation of this paper is Trajectory Convolution. Instead of summing features at the exact same spatial coordinates across time, this operation follows a "trajectory" derived from a motion field, such as optical flow.
How It Works
- Motion Tracking: For a given point, the network uses motion vectors to determine where that point was in the previous frame and where it will be in the next.
- Feature Interpolation: It then samples features from those specific "tracked" locations, rather than a static grid.
- TrajectoryNet Architecture: The authors modified a standard Separable-3D (S3D) ResNet-18, replacing the standard 1D temporal convolutions with these new trajectory convolutions.
To generate these trajectories, the system can use precomputed optical flow (like TV-L1) or MotionNet, a lightweight, "plug-and-play" CNN submodule that predicts optical flow on-the-fly. This allows the entire system to be trained end-to-end. Furthermore, concatenating the motion offsets directly with the visual features provides a minimal-cost way to give the network explicit motion cues.
Key Insights
- Motion as a First-Class Citizen: By making the convolution follow the motion, the network doesn't waste capacity learning to be invariant to translation; the architecture inherently handles the translation.
- Relation to Deformable Convolutions: Trajectory Convolution acts as a special, temporally-constrained case of 3D Deformable Convolution. While 2D deformable convolutions learn offsets to better fit object shapes, trajectory convolutions use motion to better fit temporal sequences.
- Max Pooling Preserves Motion Boundaries: When downsampling the motion field to match the feature map size deeper in the network, the authors discovered that Max Pooling significantly outperformed Average Pooling. Average pooling tends to blur trajectories, whereas max pooling preserves the sharpness of motion boundaries.
Results and Performance
The authors evaluated TrajectoryNet on two major benchmarks:
- Something-Something V1: This dataset focuses on fine-grained human-object interactions where motion is critical for distinguishing actions. TrajectoryNet showed a massive improvement here. A baseline S3D achieved 34.3% Top-1 accuracy, while TrajectoryNet (using MotionNet and motion features) jumped to 44.0%.
- Kinetics: On this massive, human-centric YouTube dataset, the model improved the S3D baseline from 76.9% to 79.8% Top-1 accuracy.
Crucially, TrajectoryNet is highly efficient. Compared to a standard "two-stream" model (which requires running two separate, full-sized networks for RGB and Flow), using precomputed flow in TrajectoryNet adds less than 10% computational overhead to the forward pass.
Limitations and Future Work
While highly effective, the approach has a few limitations. Trajectory convolution is most effective in the middle layers of the network. At very high levels, spatial resolution is too low for accurate motion fields; at very low levels, the receptive field is too small to capture meaningful motion. Additionally, while more efficient than two-stream models, the end-to-end MotionNet variant still adds parameters and latency compared to a plain 2D or (2+1)D CNN, leaving room for future work on even lighter trajectory estimation methods.
Summary updated