Is 3D Convolution with 5D Tensors Really Necessary for Video Analysis?

Habib Hajimolahoseini, Walid Ahmed, Austin Wen, Yang Liu

TL;DR

Researchers have developed a way to perform high-quality video analysis without the computational "tax" of 3D convolutions and 5D tensors. By cleverly reshaping video data into 4D tensors and using parallel 2D/1D convolution branches, their approach slashes parameter counts and FLOPs by over 50% while actually improving accuracy on standard benchmarks.

What Problem Does This Paper Address?

In the world of computer vision, moving from images (2D) to video (3D) usually means moving from 4D tensors $[Batch, Channel, Height, Width]$ to 5D tensors $[Batch, Time, Channel, Height, Width]$. While 3D convolutions are the gold standard for capturing motion, they are notoriously heavy. For context, a 3D version of ResNet can require 27 times more mathematical operations than its 2D counterpart.

Beyond the raw math, there is a hardware hurdle: many edge devices, such as those found in robotics, simply don't support 5D tensor operations or 3D convolutions efficiently. While previous researchers tried "factorizing" 3D kernels into spatial and temporal components (like the R(2+1)D architecture), many of these implementations still rely on 5D tensors under the hood. This paper asks: can we get the same results using only 4D/3D tensors and standard 2D/1D convolutions?

Methods

The authors propose a structural replacement for the standard 3D convolutional block. Instead of one heavy 3D operation, they split the task into two parallel branches:

  1. The Spatial Branch: The 5D input is reshaped into a 4D tensor by merging the Batch and Time dimensions $[B \times T, X, Y, C]$. This allows standard 2D convolutions to process every frame as an individual image.
  2. The Temporal Branch: To capture motion, the input is reshaped to merge the spatial dimensions $[B, T, X \times Y, C]$. They then apply a 2D convolution with a $d \times 1$ kernel.
  3. The "Aha" Trick: To ensure the output of the temporal branch matches the spatial branch's size (so they can be added together), they use a specific stride of $(s, s^2)$. This "square stride" guarantees that the temporal branch shrinks the data at the same rate the spatial branch does, despite the different tensor shapes.

Finally, the outputs of these branches are combined via either summation (Proposed-Add) or concatenation (Proposed-Cat).

Datasets

The team evaluated their methods using two primary video recognition benchmarks:

  • Kinetics-400: A large-scale dataset containing 400 human action classes.
  • UCF-101: A well-known dataset of 101 action categories collected from YouTube.

They used the ECO-Lite architecture as their baseline, replacing its 3D layers with their proposed 4D-compatible blocks.

Key Insights and Ideas

  • 5D is Optional: The most significant insight is that 5D tensors are a choice, not a requirement. By treating the temporal dimension as just another axis to be reshaped, we can leverage highly optimized 2D/4D hardware kernels.
  • Non-Linearity Boost: The authors found that their split-branch approach often outperformed the original 3D models. They attribute this to the increased non-linearity; by having two separate paths that are later combined, the network has more "capacity" to learn complex features without increasing the parameter count.
  • Efficiency vs. Accuracy Trade-off: The "Proposed-Add" version is the efficiency king (51% fewer parameters), while the "Proposed-Cat" version is the accuracy leader, suggesting that how you recombine spatial and temporal data is just as important as how you split them.

Key Results

The experimental results (using NVIDIA V100 GPUs) show a clear win for the 4D approach:

  • Efficiency: The "Proposed-Add" model reduced parameters from 33.7M to 16.3M and FLOPs from 35.6G to 17.1G (both ~51% reductions) compared to standard Conv3D.
  • Speed: Inference speed improved by approximately 12% in terms of Frames Per Second (FPS).
  • Accuracy: On the Kinetics dataset, the "Proposed-Cat" method achieved 59.41% accuracy, significantly outperforming the baseline Conv3D (55.63%) and even the popular R(2+1)D (57.91%).

Limitations and Future Work

While the paper demonstrates clear benefits for edge-device compatibility and raw speed, a few areas remain for exploration:

  • Kernel Assumptions: The temporal branch relies on a specific kernel reshaping that might not capture extremely complex, long-range temporal dependencies as well as a dedicated (though heavier) transformer-based attention mechanism might.
  • Architecture Specificity: The tests were primarily performed on the ECO-Lite architecture. While the logic is sound, seeing how this "4D-only" approach scales to massive models like Video Swin Transformers would be a valuable next step.
  • Hardware-Specific Benchmarking: While the motivation is edge devices (like robots), the benchmarks provided were on V100 GPUs. Future work could include power consumption and latency benchmarks on specific NPU/TPU edge hardware.

Summary updated

Abstract

In this paper, we present a comprehensive study and propose several novel techniques for implementing 3D convolutional blocks using 2D and/or 1D convolutions with only 4D and/or 3D tensors. Our motivation is that 3D convolutions with 5D tensors are computationally very expensive and they may not be supported by some of the edge devices used in real-time applications such as robots. The existing approaches mitigate this by splitting the 3D kernels into spatial and temporal domains, but they still use 3D convolutions with 5D tensors in their implementations. We resolve this issue by introducing some appropriate 4D/3D tensor reshaping as well as new combination techniques for spatial and temporal splits. The proposed implementation methods show significant improvement both in terms of efficiency and accuracy. The experimental results confirm that the proposed spatio-temporal processing structure outperforms the original model in terms of speed and accuracy using only 4D tensors with fewer parameters.
BibTeX
@misc{hajimolahoseini2024is,
  author = {Habib Hajimolahoseini and Walid Ahmed and Austin Wen and Yang Liu},
  title = {{Is 3D Convolution with 5D Tensors Really Necessary for Video Analysis?}},
  year = {2024},
  eprint = {2407.16514},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2407.16514},
}
Figure