What Can Simple Arithmetic Operations Do for Temporal Modeling?

Wenhao Wu, Yuxin Song, Zhun Sun, Jingdong Wang, Chang Xu, Wanli Ouyang

Abstract

Temporal modeling is a cornerstone of video understanding, yet it often relies on computationally expensive 3D convolutions or complex attention mechanisms. This paper introduces the Arithmetic Temporal Module (ATM), a lightweight, plug-and-play component that leverages basic arithmetic operations—addition, subtraction, multiplication, and division—between adjacent frame features to capture motion and appearance changes. Despite its simplicity, ATM achieves state-of-the-art performance on major benchmarks like Something-Something and Kinetics-400 while maintaining high computational efficiency.

Introduction

Transitioning from static image analysis to video recognition introduces the challenge of the temporal dimension. Traditionally, capturing how objects move and change over time has required heavy architectural machinery. 3D Convolutional Neural Networks (CNNs) are computationally taxing, and Vision Transformers (ViTs) often demand massive datasets and memory to learn temporal dependencies effectively.

The core research question posed here is: Do we truly need complex architectures for temporal modeling, or can simple arithmetic between frames suffice? By exploring the utility of basic mathematical operations, the authors aim to provide a more efficient path to high-performance video understanding.

The Arithmetic Temporal Module (ATM)

The ATM is designed as a parameter-free interaction layer followed by a small feature extraction block. Instead of learning complex weights to relate frames, it utilizes the inherent information in feature map differences and correlations.

Visualizing the effect of different arithmetic operations on video frame features. Subtraction acts as a proxy for motion (optical flow), while multiplication highlights correlations.

The module operates in four distinct stages:

  1. Context Spanning: For a given anchor frame $t$, the module selects neighboring context frames (e.g., $t-1, t+1$).
  2. Arithmetic Operations: It applies one or more operations between the anchor and context features:
  3. Addition (+): Captures accumulated signals and spatial occupancy.
  4. Subtraction (-): Serves as a proxy for optical flow, highlighting motion tendencies.
  5. Multiplication (×): Measures similarity or correlation between regions across frames.
  6. Division (÷): Standardizes and amplifies subtle changes.
  7. Feature Extraction: A small stack of 2D convolutions processes these "arithmetic cues" to distill meaningful temporal patterns.
  8. Domain Transformation: The resulting features are projected back to the original dimensions and integrated into the main backbone via a residual connection.

Key Insights and Results

The authors evaluated ATM on several diverse video recognition benchmarks, including the motion-heavy Something-Something (SS) V1 & V2 and the appearance-focused Kinetics-400.

Performance Highlights

  • State-of-the-Art Accuracy: On Something-Something V2, ATM reached 74.6% Top-1 accuracy using a ViT-L backbone, outperforming specialized models like CoVeR and UniFormerV2.
  • Efficiency Gains: ATM-equipped models show remarkable efficiency. For instance, on SS V1, an ATM-ResNet-18 outperformed a much larger TSM-ResNet-50 (48.7% vs 47.1% accuracy) with significantly lower computational overhead.
  • The Power of Subtraction and Multiplication: Ablation studies identified subtraction (motion) and multiplication (correlation) as the most effective temporal cues. Combining these two operations generally yielded the best performance across tasks.

Architectural Flexibility

ATM is inherently flexible. It can be inserted into standard CNNs (like ResNet) or Transformers (like ViT) with minimal modification. The researchers found that placing ATM in the middle layers of a network (e.g., after the 7th layer of a 12-layer ViT) is more effective than at the extreme ends, as it benefits from mid-level semantic features.

Conclusion

The Arithmetic Temporal Module demonstrates that sophisticated temporal modeling does not strictly require complex, high-parameter architectures. By returning to the fundamentals of arithmetic, ATM provides a powerful, efficient, and easy-to-implement solution for video understanding. This work suggests that the simplest mathematical relationships between frames are often the most robust indicators of motion and change.

Summary updated

Abstract

Temporal modeling plays a crucial role in understanding video content. To tackle this problem, previous studies built complicated temporal relations through time sequence thanks to the development of computationally powerful devices. In this work, we explore the potential of four simple arithmetic operations for temporal modeling. Specifically, we first capture auxiliary temporal cues by computing addition, subtraction, multiplication, and division between pairs of extracted frame features. Then, we extract corresponding features from these cues to benefit the original temporal-irrespective domain. We term such a simple pipeline as an Arithmetic Temporal Module (ATM), which operates on the stem of a visual backbone with a plug-and-play style. We conduct comprehensive ablation studies on the instantiation of ATMs and demonstrate that this module provides powerful temporal modeling capability at a low computational cost. Moreover, the ATM is compatible with both CNNs- and ViTs-based architectures. Our results show that ATM achieves superior performance over several popular video benchmarks. Specifically, on Something-Something V1, V2 and Kinetics-400, we reach top-1 accuracy of 65.6%, 74.6%, and 89.4% respectively. The code is available at https://github.com/whwu95/ATM.
BibTeX
@inproceedings{wu2023what,
  author = {Wenhao Wu and Yuxin Song and Zhun Sun and Jingdong Wang and Chang Xu and Wanli Ouyang},
  title = {{What Can Simple Arithmetic Operations Do for Temporal Modeling?}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2023},
  doi = {10.1109/iccv51070.2023.01261},
}
Figure