Abstract
Various convolutional neural networks (CNNs) were developed recently that achieved accuracy comparable with that of human beings in computer vision tasks such as image recognition, object detection and tracking, etc. Most of these networks, however, process one single frame of image at a time, and may not fully utilize the temporal and contextual correlation typically present in multiple channels of the same image or adjacent frames from a video, thus limiting the achievable throughput. This limitation stems from the fact that existing CNNs operate on deterministic numbers. In this paper, we propose a novel statistical convolutional neural network (SCNN), which extends existing CNN architectures but operates directly on correlated distributions rather than deterministic numbers. By introducing a parameterized canonical model to model correlated data and defining corresponding operations as required for CNN training and inference, we show that SCNN can process multiple frames of correlated images effectively, hence achieving significant speedup over existing CNN models. We use a CNN based video object detection as an example to illustrate the usefulness of the proposed SCNN as a general network model. Experimental results show that even a non-optimized implementation of SCNN can still achieve 178% speedup over existing CNNs with slight accuracy degradation.
Introduction
In the realm of computer vision, Convolutional Neural Networks (CNNs) have set the standard for performance. However, a significant efficiency bottleneck remains in video processing: the "tunnel vision" problem. Most architectures process video one frame at a time, treating each image as a collection of deterministic, static numbers. This approach largely ignores the massive amount of redundant information shared between adjacent video frames. In real-time applications like drone-based surveillance or autonomous driving, re-calculating features for nearly identical frames (where frame $N$ and $N+1$ are often 99% similar) leads to unnecessary computational overhead.
To address this, we propose the Statistical Convolutional Neural Network (SCNN). SCNN shifts the paradigm from processing pixels to processing distributions, allowing a single forward pass to effectively analyze multiple frames simultaneously.
Methodology: Thinking Statistically
The core innovation of SCNN is the replacement of deterministic pixel values with parameterized statistical distributions that model temporal correlations.
Canonical Model Extraction
The framework utilizes Independent Component Analysis (ICA) to model a sequence of frames, or a "snippet," as a linear additive model. Each pixel is represented in a parameterized canonical form, consisting of a mean value and a weighted sum of shared random components that capture temporal variance.
Statistical Layers
SCNN redefines standard CNN operations to function directly on these distributions:
- Linear Layers (Convolutional, Fully Connected): These are updated to operate on the distribution coefficients.
- Non-linear Layers (ReLU, Max-pooling): These layers utilize analytical approximations, such as "tightness probabilities," to estimate the maximum of multiple Gaussian distributions. This ensures the network remains differentiable and capable of end-to-end training via backpropagation.
Additionally, a Polynomial Fitting Loss is introduced. This objective function uses least-squares polynomial fitting to ensure that predicted object trajectories remain smooth across the video snippet, further leveraging temporal context.
Key Results
The SCNN framework was evaluated using the DAC-Contest 2018 dataset, a challenging benchmark featuring drone-captured footage with small objects and rapid movement. We tested SCNN as a backbone for several popular architectures, including VGG and ResNet variants.
The results demonstrate a significant performance-efficiency trade-off:
- Throughput Gains: Using a VGG16 backbone, SCNN achieved a 178% speedup, increasing throughput from 65 FPS to 116 FPS. On ResNet18, the system reached a blistering 363 FPS, compared to the standard 247 FPS.
- Accuracy Trade-off: While there is a slight decrease in Mean Average Precision (mAP)—for instance, VGG16 dropped from 61.9% to 58.3%—the gains in speed are highly favorable for real-time systems.
- Implicit Robustness: SCNN showed resilience in scenarios with partial occlusions or lens flares, as the ICA modeling implicitly "smooths" transient noise by considering the broader temporal context.
Conclusion
SCNN offers a promising path toward "statistical" deep learning. By moving away from brute-force deterministic processing toward a distribution-aware approach, SCNN achieves substantial speedups in video object detection. Future work will explore non-linear parameterized distributions to better capture complex spatial correlations and low-level CUDA optimizations to further enhance throughput.
Summary updated