GPU-FV: Realtime Fisher Vector and Its Applications in Video Monitoring

Wenjing Ma, Liangliang Cao, Lei Yu, Guoping Long, Yucheng Li

Abstract

Fisher vector has been widely used in many multimedia retrieval and visual recognition applications with good performance. However, the computation complexity prevents its usage in real-time video monitoring. In this work, we proposed and implemented GPU-FV, a fast Fisher vector extraction method with the help of modern GPUs. The challenge of implementing Fisher vector on GPUs lies in the data dependency in feature extraction and expensive memory access in Fisher vector computing. To handle these challenges, we carefully designed GPU-FV in a way that utilizes the computing power of GPU as much as possible, and applied optimizations such as loop tiling to boost the performance. GPU-FV is about 12 times faster than the CPU version, and 50% faster than a non-optimized GPU implementation. For standard video input (320*240), GPU-FV can process each frame within 34ms on a model GPU. Our experiments show that GPU-FV obtains a similar recognition accuracy as traditional FV on VOC 2007 and Caltech 256 image sets. We also applied GPU-FV for realtime video monitoring tasks and found that GPU-FV outperforms a number of previous works. Especially, when the number of training examples are small, GPU-FV outperforms the recent popular deep CNN features borrowed from ImageNet. The code can be downloaded from the following link https://bitbucket.org/mawenjing/gpu-fv.

Introduction

While Deep Convolutional Neural Networks (CNNs) currently dominate large-scale visual recognition, they require massive amounts of training data to achieve high performance. For scenarios with small-to-medium datasets, Fisher Vectors (FVs) remain a highly accurate and attractive alternative. However, extracting FVs is computationally heavy—typically taking 2 to 5 seconds per image on a standard CPU. This latency disqualifies FVs from real-time applications like live video surveillance, anomaly detection, and expression tracking.

The authors address this bottleneck by introducing "GPU-FV," a redesigned FV extraction pipeline specifically optimized for modern GPU architectures. By accelerating both Dense SIFT feature extraction and Gaussian Mixture Model (GMM) estimation, the system achieves real-time performance without sacrificing accuracy.

Methodology and Optimizations

Naive porting of CPU code to GPUs is often inefficient due to differences in architecture. The authors introduced several novel algorithmic tweaks to maximize GPU utilization:

Redesigning Dense SIFT

Standard CPU implementations use a 4-step sequential convolution that is hostile to GPU parallelization because it requires scan operations with subtraction. The authors mathematically combined these into a 2-step addition-only process, making it highly parallelizable.

Restructuring Dense SIFT for GPU parallelization

Memory and Computation Management

FV encoding was split into two phases: calculating posteriors and accumulating the final vector. To prevent bandwidth bottlenecks, the implementation heavily utilizes GPU shared memory. Specific optimizations include:

  • Loop Tiling: Operations were grouped to reduce thread synchronization overhead by 75%.
  • Early Termination: By recognizing that GMM posterior values are often extremely sparse, the system skips calculations if a posterior value falls below a certain threshold.
  • Vectorization: Multiple data elements are packed into single vector operations (SIMD).
  • Scale Approximation: Dropping the largest image scale during Dense SIFT extraction (using 8 scales instead of 9) drastically reduced computation time with negligible impact on accuracy.

Experimental Results

The performance of GPU-FV was validated across standard image classification datasets (PASCAL VOC 2007 and Caltech 256) and video monitoring tasks (UMN Crowd Activity and a custom Baby Laughing dataset).

Speed and Efficiency

GPU-FV processes a standard 320x240 video frame in 34ms (~29 FPS) on an NVIDIA Tesla K40. As shown in the performance comparison, this is roughly 12.7x faster than a single-threaded CPU and 4.5x faster than a 16-thread CPU implementation.

Performance comparison across hardware and scales

Accuracy and Real-world Application

On VOC 2007, the 8-scale GPU-FV achieved 59.3% mAP, nearly identical to the CPU baseline (59.93%). In niche video tasks where training data is scarce, GPU-FV significantly outperformed off-the-shelf CNN features:

  • Crowd Activity: On the UMN dataset, GPU-FV achieved an AUC of 0.984, beating Deep CNN features (0.930) and C3D (0.946).
  • Expression Detection: For the baby laughing task, GPU-FV achieved an AUC of 0.935, significantly outperforming the AlexNet/Caffe baseline (0.655).

Conclusion

GPU-FV demonstrates that Fisher Vectors can be viable for real-time video monitoring through careful algorithmic restructuring and GPU-specific optimizations. While currently benchmarked at 320x240 resolution, the approach offers a powerful alternative to CNNs in low-data regimes. Future work will focus on scaling the algorithm to higher resolutions and broader video processing applications.

Summary updated

Abstract

Fisher vector has been widely used in many multimedia retrieval and visual recognition applications with good performance. However, the computation complexity prevents its usage in real-time video monitoring. In this work, we proposed and implemented GPU-FV, a fast Fisher vector extraction method with the help of modern GPUs. The challenge of implementing Fisher vector on GPUs lies in the data dependency in feature extraction and expensive memory access in Fisher vector computing. To handle these challenges, we carefully designed GPU-FV in a way that utilizes the computing power of GPU as much as possible, and applied optimizations such as loop tiling to boost the performance. GPU-FV is about 12 times faster than the CPU version, and 50\% faster than a non-optimized GPU implementation. For standard video input (320*240), GPU-FV can process each frame within 34ms on a model GPU. Our experiments show that GPU-FV obtains a similar recognition accuracy as traditional FV on VOC 2007 and Caltech 256 image sets. We also applied GPU-FV for realtime video monitoring tasks and found that GPU-FV outperforms a number of previous works. Especially, when the number of training examples are small, GPU-FV outperforms the recent popular deep CNN features borrowed from ImageNet. The code can be downloaded from the following link https://bitbucket.org/mawenjing/gpu-fv.
BibTeX
@inproceedings{ma2016gpu,
  author = {Wenjing Ma and Liangliang Cao and Lei Yu and Guoping Long and Yucheng Li},
  title = {{GPU-FV: Realtime Fisher Vector and Its Applications in Video Monitoring}},
  booktitle = {ICMR},
  year = {2016},
  url = {https://openreview.net/forum?id=cEG1GMx3tE6},
}
Figure