SGE NET: Video Object Detection with Squeezed GRU and Information Entropy Map

Rui Su, Wenjing Huang, Haoyu Ma, Xiaowei Song, Jinglu Hu

TL;DR

SGE-Net is a highly efficient video object detection framework that achieves a 3.7% mAP improvement over baseline models while slashing temporal module parameters by nearly 90%. It achieves this by introducing a "Squeezed GRU" for lightweight temporal modeling and a novel 2D Information Entropy map that provides global semantic cues without adding any learnable parameters.

What Problem Does This Paper Address?

Object detection in video is significantly more complex than in static images. While videos provide rich temporal context, they also introduce challenges like motion blur, occlusion, and varying object appearances.

Current state-of-the-art methods often use Recurrent Neural Networks (RNNs) like GRUs or LSTMs to bridge the gap between frames. However, these modules are computationally expensive and parameter-heavy, making them difficult to deploy in real-time scenarios like robotics or mobile surveillance. Furthermore, standard Convolutional Neural Networks (CNNs) suffer from limited receptive fields, often missing the "big picture" (global context) of a frame. The authors sought a way to capture both temporal correlations and global spatial cues without the massive computational overhead typical of recurrent architectures.

Methods

The authors introduce two primary innovations integrated into a framework called SGE-Net:

  1. Squeezed GRU: Traditional GRUs use Hadamard products that treat images as vectors, losing spatial correlation. While "ConvGRU" fixes this by using convolutions, it remains parameter-heavy. The Squeezed GRU optimizes this by:
  2. Replacing standard convolutions with depth-wise separable convolutions.
  3. Implementing channel reduction (using $1 \times 1$ convolutions) to shrink the feature space before processing.
  4. This allows the model to maintain spatial-temporal correlations with only ~10% of the parameters of a standard GRU.
  5. 2D Information Entropy (IE) Map: Traditional entropy provides a single global value for an image, which lacks spatial detail. The authors propose a 2D IE map calculated via a sliding window.
  6. It captures the relationship between a pixel's gray value and the average gray value of its neighbors.
  7. Because this is a deterministic calculation based on the input image, it acts as a "free" attention mechanism that helps the network locate and classify objects based on their unique entropy distributions—without adding a single learnable parameter.

The overall architecture uses a MobileNet-SSD backbone, inserts the IE map for feature enhancement, and places the Squeezed GRU at the end of the feature extraction phase to model temporal flow.

Datasets

The model was evaluated on the ImageNet VID 2015 dataset, a standard benchmark for video object detection.

  • Training: 3,862 video snippets.
  • Testing: 555 video snippets across 30 object categories.
  • Pre-training: The network was initialized with weights from the standard ImageNet classification dataset.

Key Insights and Ideas

  • Entropy as "Zero-Cost" Attention: One of the most elegant ideas in the paper is using traditional image processing (entropy) to supplement deep learning. Since different objects and environments have distinct entropy signatures, the IE map provides a semantic "roadmap" that helps the CNN focus its attention.
  • The "Squeeze" Doesn't Sacrifice Accuracy: The authors proved that you don't need millions of parameters to model temporal data in video. By aggressively reducing channels and using depth-wise operations, they achieved performance comparable to (or better than) heavy ConvGRU models.
  • High-Level Placement is Key: Through ablation studies, the authors found that both the Squeezed GRU and the IE map perform best when applied to high-level feature maps (like the final layer of the backbone). Applying them to low-level, high-resolution maps actually degraded performance, likely because those layers lack the semantic density needed to make sense of global entropy or temporal flow.

Key Results

  • Accuracy: SGE-Net achieved a 46.8% mAP, a significant jump from the 43.1% baseline (MNet-SSD).
  • Efficiency: The Squeezed GRU module contains only 0.67 million parameters, compared to 6.33 million in a standard GRU and 29.49 million in a ConvGRU.
  • Computational Savings: Compared to a standard ConvGRU, SGE-Net saves over 40% in computational costs (measured in Giga-MACs), making it much more suitable for real-time applications.
  • IE Map Impact: Adding the IE map alone provided a 1.1% mAP boost over the Squeezed GRU baseline with zero increase in the parameter count.

Limitations and Future Work

The authors noted that the IE map's effectiveness is sensitive to where it is placed in the network; it is currently optimized for high-level semantic features and may not be as useful for tasks requiring fine-grained low-level details.

For future work, the authors suggest:

  • Applying the Squeezed GRU and IE map modules to other video-based tasks, such as video semantic segmentation and multi-person pose estimation.
  • Exploring different types of information-theoretic descriptors beyond 2D entropy to further enhance feature representation.

Summary updated

Abstract

Recently, deep learning based video object detection has attracted more and more attention. Compared with object detection of static images, video object detection is more challenging due to the motion of objects, while providing rich temporal information. The RNN-based algorithm is an effective way to enhance detection performance in videos with temporal information. However, most studies in this area only focus on accuracy while ignoring the calculation cost and the number of parameters. In this paper, we propose an efficient method that combines channel-reduced convolutional GRU (Squeezed GRU), and Information Entropy map for video object detection (SGE-Net). The experimental results validate the accuracy improvement, computational savings of the Squeezed GRU, and superiority of the information entropy attention mechanism on the classification performance. The mAP has increased by 3.7 contrasted with the baseline, and the number of parameters has decreased from 6.33 million to 0.67 million compared with the standard GRU.
BibTeX
@inproceedings{su2021sge,
  author = {Rui Su and Wenjing Huang and Haoyu Ma and Xiaowei Song and Jinglu Hu},
  title = {{SGE NET: Video Object Detection with Squeezed GRU and Information Entropy Map}},
  booktitle = {ICIP},
  year = {2021},
  doi = {10.1109/icip42928.2021.9506081},
}

Loading related content...

Figure