SCNN: A General Distribution Based Statistical Convolutional Neural Network with Application to Video Object Detection

Tianchen Wang, Jinjun Xiong, Xiaowei Xu, Yiyu Shi

Introduction

Standard Convolutional Neural Networks (CNNs) are primarily designed for static image processing. When applied to video sequences, these networks typically process each frame independently as a deterministic set of pixel values. This approach is computationally expensive and fundamentally inefficient, as it ignores the significant temporal and contextual correlations between adjacent frames. While techniques like 3D convolutions or optical flow have been proposed to address this, they often introduce substantial complexity and computational overhead.

Statistical Convolutional Neural Networks (SCNN) offer a paradigm shift by reimagining video processing through the lens of statistical distributions. Instead of treating a video as a sequence of independent images, SCNN models a "snippet" of frames as a single entity characterized by correlated distributions. By leveraging Independent Component Analysis (ICA) to model temporal redundancy, SCNN can process multiple frames in a single forward pass, achieving significant speedups with minimal impact on detection accuracy.

Methodology

The core innovation of SCNN lies in transforming the underlying deterministic math of a CNN into a statistical framework. This transformation involves three primary components: the canonical model, feature extraction via ICA, and the redesign of standard neural layers.

Linear Parameterized Canonical Model

In SCNN, a pixel is no longer represented by a single scalar value. Instead, it is modeled as a distribution: $$D_i = a_{i,0} + \sum a_{i,k}X_k + a_{i,r}R_i$$ In this equation, $X_k$ represents independent random subcomponents (the "basis") that capture shared information across the frame sequence, while $a_{i,k}$ are the parameters that the network processes. This representation allows the network to handle the temporal information of $N$ frames within a single parameterized structure.

ICA for Feature Extraction and Layer Redesign

The system uses Independent Component Analysis (ICA) to extract these basis distributions from a snippet of $N$ frames. This compresses the temporal data into a manageable set of basis dimensions ($m$).

The SCNN architecture, illustrating the extraction of basis distributions via ICA and the subsequent neural network processing using distribution parameters.

To maintain this statistical representation throughout the network, standard operations—including Convolution, ReLU, Max-Pooling, and Batch Normalization—were mathematically redefined. A significant challenge was the non-linear "Max" operation required for ReLU and Pooling. The authors addressed this by adapting techniques from Statistical Static Timing Analysis (SSTA)—a field in hardware chip design—to approximate the maximum of distributions using "tightness probabilities."

Experimental Results

The SCNN framework was evaluated using the DAC-Contest 2018 dataset, an industrial UAV (Unmanned Aerial Vehicle) dataset featuring small objects (cars, boats, people) moving through diverse environments. The SCNN backbone was integrated into a YOLOv2-style detector, utilizing a polynomial fitting loss to ensure smooth object trajectories across frames.

Performance and Efficiency

The primary advantage of SCNN is its massive increase in processing speed. The theoretical speedup factor is approximately $N/m$, where $N$ is the snippet size and $m$ is the number of basis dimensions. In practice, SCNN achieved up to a 178% speedup over conventional CNNs. For instance, a VGG16-based model improved from 65 FPS to 116 FPS.

Performance comparison across various backbones, showing the trade-off between mean Average Precision (mAP) and frames per second (FPS) as the number of basis components (m) varies.

As shown in the table above, there is a clear trade-off between the number of basis components ($m$) and accuracy. On a VGG11 backbone, reducing $m$ to 8 nearly doubles the frame rate (from 101 to 181 FPS) while the mean Average Precision (mAP) only drops from 62.2% to 56.5%. Interestingly, in categories with smooth temporal transitions like "car" or "riding," SCNN occasionally outperformed standard frame-by-frame CNNs due to its implicit temporal smoothing.

Conclusion and Future Work

SCNN demonstrates that modeling video data statistically rather than deterministically can yield dramatic efficiency gains. By processing distribution parameters, the network effectively performs the work of multiple frames simultaneously. While the current model relies on a linear canonical form, future research may explore non-linear models to capture more complex spatial correlations. Additionally, further optimization of the ICA preprocessing step and the implementation of dedicated CUDA kernels could push the performance boundaries even further, making SCNN a powerful tool for real-time video analytics on edge devices.

Summary updated

Abstract

Various convolutional neural networks (CNNs) were developed recently that achieved accuracy comparable with that of human beings in computer vision tasks such as image recognition, object detection and tracking, etc. Most of these networks, however, process one single frame of image at a time, and may not fully utilize the temporal and contextual correlation typically present in multiple channels of the same image or adjacent frames from a video, thus limiting the achievable throughput. This limitation stems from the fact that existing CNNs operate on deterministic numbers. In this paper, we propose a novel statistical convolutional neural network (SCNN), which extends existing CNN architectures but operates directly on correlated distributions rather than deterministic numbers. By introducing a parameterized canonical model to model correlated data and defining corresponding operations as required for CNN training and inference, we show that SCNN can process multiple frames of correlated images effectively, hence achieving significant speedup over existing CNN models. We use a CNN based video object detection as an example to illustrate the usefulness of the proposed SCNN as a general network model. Experimental results show that even a nonoptimized implementation of SCNN can still achieve 178% speedup over existing CNNs with slight accuracy degradation.
BibTeX
@inproceedings{wang2019scnn,
  author = {Tianchen Wang and Jinjun Xiong and Xiaowei Xu and Yiyu Shi},
  title = {{SCNN: A General Distribution Based Statistical Convolutional Neural Network with Application to Video Object Detection}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2019},
  doi = {10.1609/aaai.v33i01.33015321},
}
Figure