Abstract
Correspondences between frames encode rich information about dynamic content in videos. However, it is challenging to effectively capture and learn those due to their irregular structure and complex dynamics. In this paper, we propose a novel neural network that learns video representations by aggregating information from potential correspondences. This network, named CPNet, can learn evolving 2D fields with temporal consistency. In particular, it can effectively learn representations for videos by mixing appearance and long-range motion with an RGB-only input. CPNet shows stronger performance than existing methods on Kinetics and achieves the state-of-the-art performance on Something-Something and Jester.
Introduction
Understanding video requires more than just recognizing objects in static frames; it requires capturing how those objects move and interact over time. Traditional 2D CNNs excel at static features but struggle with temporal dynamics. While 3D CNNs and "Non-Local" (NL) networks attempt to bridge this gap, they face significant hurdles. 3D CNNs often rely on stacking local operations, making it computationally expensive to capture long-range motion. Non-Local Networks capture global context but often ignore specific positional information—where things moved from and to—which is crucial for understanding complex actions. Optical Flow methods provide motion information but require a separate, expensive pre-computation branch. CPNet seeks to learn these "correspondences" directly within a single-stream RGB network.
The CPNet Architecture
The core innovation is the CP (Correspondence Proposal) Module, which can be inserted into standard architectures like ResNet. It treats video features as a "point cloud" to efficiently capture long-range temporal dynamics.
The module operates in three main steps:
- k-NN Grouping: The module treats the video representation tensor as a point cloud of features. For every feature point, it searches all other frames to find the $k$ most similar features in "semantic space" using $L_2$ distance.
- Correspondence Embedding: For each of these $k$ potential matches, the module creates a triplet consisting of the original feature, the candidate feature, and their relative spatiotemporal displacement $(\Delta t, \Delta h, \Delta w)$. This triplet is fed into a shared Multi-Layer Perceptron (MLP).
- Aggregation: A max-pooling operation is applied across the $k$ proposals. This allows the network to "pick" the most informative correspondence and ignore noisy or incorrect proposals.
By including the $(\Delta t, \Delta h, \Delta w)$ displacement, the network doesn't just see that two things look alike; it explicitly learns the "reason" for the selection based on how the feature moved through space and time.
Key Insights
- The Point Cloud Analogy: Borrowing from 3D deep learning (specifically PointNet), the authors realized that video motion is essentially an irregular set of correspondences. Treating features as unordered points allows the model to bypass the rigid grid constraints of standard convolutions.
- Sparsity and Efficiency: Unlike Non-Local blocks that compute relationships between every pair of pixels ($O(N^2)$), CPNet only looks at the top $k$ candidates. This focus saves memory and computation while allowing for more complex reasoning.
- Robustness via Max-Pooling: The $k$-NN search might return "wrong" proposals. The max-pooling layer acts as a filter, ensuring that only the most relevant motion signals affect the final representation.
- Positional Reasoning: Most "appearance-only" models fail on datasets like Something-Something because they can't distinguish between "pulling" and "pushing." By encoding $\Delta h$ and $\Delta w$, CPNet learns the directionality of motion directly.
Evaluation and Results
The authors evaluated CPNet on several major benchmarks, including Kinetics-400, Something-Something v2, and Jester v1.
On the motion-heavy Something-Something v2 dataset, CPNet (ResNet-34) achieved 57.65% top-1 accuracy, outperforming the Two-stream TRN while using significantly fewer parameters. It also reached state-of-the-art performance on Jester v1 with 96.7% top-1 accuracy. On Kinetics-400, CPNet (ResNet-101) achieved 75.3% top-1, beating the Non-Local C2D baseline while being more parameter-efficient (42.1M vs 48.2M parameters).
Ablation studies confirmed that the accuracy gain from adding CP modules is significant (up to 14% on some datasets) while the increase in model size remains negligible.
Conclusion
CPNet demonstrates that explicitly modeling correspondences as proposals in a semantic feature space is a powerful and efficient way to learn video representations. By treating video as a point cloud and focusing on the most relevant spatiotemporal displacements, the model achieves state-of-the-art results on challenging motion-centric datasets using only RGB input. Future work may extend this module to tasks like video object detection and tracking.
Summary updated