PASS: Patch Automatic Skip Scheme for Efficient Real-Time Video Perception on Edge Devices

Qihua Zhou, Song Guo, Jun Pan, Jiacheng Liang, Zhenda Xu, Jingren Zhou

Abstract

Real-time video perception on edge devices often requires a compromise between accuracy and speed. This paper introduces PASS (Patch Automatic Skip Scheme), a plug-and-play acceleration framework that identifies and skips redundant image patches in video streams. By utilizing a lightweight, self-supervised "gate" module, PASS exploits temporal redundancy at the patch level. Experimental results on human pose estimation and object tracking benchmarks demonstrate up to 12x speedups on edge hardware like the NVIDIA Jetson Nano while maintaining state-of-the-art accuracy.

Introduction

Video data is inherently redundant; between consecutive frames, most pixels remain unchanged. Current efficiency methods typically rely on keyframe selection, which can cause accuracy jitter, or model compression, which is often task-specific. PASS addresses these limitations by providing a task-independent method to reduce computation on commodity hardware. Instead of skipping entire frames, PASS skips specific patches that have not changed significantly enough to affect high-level feature extraction.

The PASS framework enables plug-and-play acceleration for general usage across diverse downstream tasks like human pose estimation and object tracking.

Methodology

The core of PASS is the Binary Mask Gate, a tiny neural module inserted before convolutional layers. It operates through a three-step process:

  1. Patch Embedding: The input frame is divided into non-overlapping $n \times n$ squares.
  2. Gate Architecture: The gate processes current and previous features through a lightweight stack of depth-wise separable convolutions.
  3. Differentiable Decisions: Using Gumbel Normalization and Bernoulli sampling, the gate outputs a binary mask where a "1" indicates a redundant patch to be skipped (reusing the previous feature) and a "0" indicates a patch requiring computation.
Detailed architecture of the Binary Mask Gate, showing the patch embedding, convolutional stack, and patch classifier components.

Training Strategy

PASS employs a two-stage training approach. First, Self-Supervised Pre-training uses a triplet loss to teach the gate to distinguish between similar and different patches without human labels. Second, Task-Oriented Fine-tuning allows the backbone network to adapt to the sparse feature maps generated by the skipping process.

Experimental Results

The researchers evaluated PASS on heavy-duty benchmarks including Human3.6M and MPII for pose estimation, and MOTChallenge for object tracking. Testing on an NVIDIA Jetson Nano revealed substantial performance gains:

  • 3D Human Pose Estimation: PASS achieved a 9.43x speedup and reduced Multiply-Accumulate (MAC) operations by 62.61%, while slightly improving accuracy (lower MPJPE).
  • Multiple Object Tracking: The system reached a 12.19x speedup and a 73.16% MAC reduction compared to the FairMOT baseline.
  • Environmental Robustness: In high-motion scenarios, PASS dynamically adapts its skip strategy, maintaining high tracking accuracy where other sparse models often fail.

Discussion and Conclusion

A key insight of PASS is that even foreground elements can be redundant if their motion is minimal relative to the task. By decoupling acceleration from specific tasks, PASS can be integrated into various backbones like HRNet or ResNet. Furthermore, patch-level skipping leverages block-wise sparse convolution, which is natively supported by modern edge GPU and CPU instructions.

While highly effective, future work could explore multi-scale patch sizes for high-resolution video and further optimize gate overhead in extremely deep networks. PASS demonstrates that the most efficient way to process video is to work smarter by identifying and ignoring redundant information.

Summary updated

Abstract

Real-time video perception tasks are often challenging over the resource-constrained edge devices due to the concerns of accuracy drop and hardware overhead, where saving computations is the key to performance improvement. Existing methods either rely on domain-specific neural chips or priorly searched models, which require specialized optimization according to different task properties. In this work, we propose a general and task-independent Patch Automatic Skip Scheme (PASS), a novel end-to-end learning pipeline to support diverse video perception settings by decoupling acceleration and tasks. The gist is to capture the temporal similarity across video frames and skip the redundant computations at patch level, where the patch is a non-overlapping square block in visual. PASS equips each convolution layer with a learnable gate to selectively determine which patches could be safely skipped without degrading model accuracy. As to each layer, a desired gate needs to make flexible skip decisions based on intermediate features without any annotations, which cannot be achieved by conventional supervised learning paradigm. To address this challenge, we are the first to construct a tough self-supervisory procedure for optimizing these gates, which learns to extract contrastive representation, i.e., distinguishing similarity and difference, from frame sequence. These high-capacity gates can serve as a plug-and-play module for convolutional neural network (CNN) backbones to implement patch-skippable architectures, and automatically generate proper skip strategy to accelerate different video-based downstream tasks, e.g., outperforming the state-of-the-art MobileHumanPose (MHP) in 3D pose estimation and FairMOT in multiple object tracking, by up to 9.43 times and 12.19 times speedups, respectively. By directly processing the raw data of frames, PASS can generalize to real-time video streams on commodity edge devices, e.g., NVIDIA Jetson Nano, with efficient performance in realistic deployment.
BibTeX
@inproceedings{zhou2023pass,
  author = {Qihua Zhou and Song Guo and Jun Pan and Jiacheng Liang and Zhenda Xu and Jingren Zhou},
  title = {{PASS: Patch Automatic Skip Scheme for Efficient Real-Time Video Perception on Edge Devices}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2023},
  doi = {10.1609/aaai.v37i3.25491},
}
Figure