Abstract
Temporal modeling is a cornerstone of video understanding, yet it often relies on computationally expensive 3D convolutions or complex attention mechanisms. This paper introduces the Arithmetic Temporal Module (ATM), a lightweight, plug-and-play component that leverages basic arithmetic operations—addition, subtraction, multiplication, and division—between adjacent frame features to capture motion and appearance changes. Despite its simplicity, ATM achieves state-of-the-art performance on major benchmarks like Something-Something and Kinetics-400 while maintaining high computational efficiency.
Introduction
Transitioning from static image analysis to video recognition introduces the challenge of the temporal dimension. Traditionally, capturing how objects move and change over time has required heavy architectural machinery. 3D Convolutional Neural Networks (CNNs) are computationally taxing, and Vision Transformers (ViTs) often demand massive datasets and memory to learn temporal dependencies effectively.
The core research question posed here is: Do we truly need complex architectures for temporal modeling, or can simple arithmetic between frames suffice? By exploring the utility of basic mathematical operations, the authors aim to provide a more efficient path to high-performance video understanding.
The Arithmetic Temporal Module (ATM)
The ATM is designed as a parameter-free interaction layer followed by a small feature extraction block. Instead of learning complex weights to relate frames, it utilizes the inherent information in feature map differences and correlations.
The module operates in four distinct stages:
- Context Spanning: For a given anchor frame $t$, the module selects neighboring context frames (e.g., $t-1, t+1$).
- Arithmetic Operations: It applies one or more operations between the anchor and context features:
- Addition (+): Captures accumulated signals and spatial occupancy.
- Subtraction (-): Serves as a proxy for optical flow, highlighting motion tendencies.
- Multiplication (×): Measures similarity or correlation between regions across frames.
- Division (÷): Standardizes and amplifies subtle changes.
- Feature Extraction: A small stack of 2D convolutions processes these "arithmetic cues" to distill meaningful temporal patterns.
- Domain Transformation: The resulting features are projected back to the original dimensions and integrated into the main backbone via a residual connection.
Key Insights and Results
The authors evaluated ATM on several diverse video recognition benchmarks, including the motion-heavy Something-Something (SS) V1 & V2 and the appearance-focused Kinetics-400.
Performance Highlights
- State-of-the-Art Accuracy: On Something-Something V2, ATM reached 74.6% Top-1 accuracy using a ViT-L backbone, outperforming specialized models like CoVeR and UniFormerV2.
- Efficiency Gains: ATM-equipped models show remarkable efficiency. For instance, on SS V1, an ATM-ResNet-18 outperformed a much larger TSM-ResNet-50 (48.7% vs 47.1% accuracy) with significantly lower computational overhead.
- The Power of Subtraction and Multiplication: Ablation studies identified subtraction (motion) and multiplication (correlation) as the most effective temporal cues. Combining these two operations generally yielded the best performance across tasks.
Architectural Flexibility
ATM is inherently flexible. It can be inserted into standard CNNs (like ResNet) or Transformers (like ViT) with minimal modification. The researchers found that placing ATM in the middle layers of a network (e.g., after the 7th layer of a 12-layer ViT) is more effective than at the extreme ends, as it benefits from mid-level semantic features.
Conclusion
The Arithmetic Temporal Module demonstrates that sophisticated temporal modeling does not strictly require complex, high-parameter architectures. By returning to the fundamentals of arithmetic, ATM provides a powerful, efficient, and easy-to-implement solution for video understanding. This work suggests that the simplest mathematical relationships between frames are often the most robust indicators of motion and change.
Summary updated