Abstract
Real-time video perception on edge devices often requires a compromise between accuracy and speed. This paper introduces PASS (Patch Automatic Skip Scheme), a plug-and-play acceleration framework that identifies and skips redundant image patches in video streams. By utilizing a lightweight, self-supervised "gate" module, PASS exploits temporal redundancy at the patch level. Experimental results on human pose estimation and object tracking benchmarks demonstrate up to 12x speedups on edge hardware like the NVIDIA Jetson Nano while maintaining state-of-the-art accuracy.
Introduction
Video data is inherently redundant; between consecutive frames, most pixels remain unchanged. Current efficiency methods typically rely on keyframe selection, which can cause accuracy jitter, or model compression, which is often task-specific. PASS addresses these limitations by providing a task-independent method to reduce computation on commodity hardware. Instead of skipping entire frames, PASS skips specific patches that have not changed significantly enough to affect high-level feature extraction.
Methodology
The core of PASS is the Binary Mask Gate, a tiny neural module inserted before convolutional layers. It operates through a three-step process:
- Patch Embedding: The input frame is divided into non-overlapping $n \times n$ squares.
- Gate Architecture: The gate processes current and previous features through a lightweight stack of depth-wise separable convolutions.
- Differentiable Decisions: Using Gumbel Normalization and Bernoulli sampling, the gate outputs a binary mask where a "1" indicates a redundant patch to be skipped (reusing the previous feature) and a "0" indicates a patch requiring computation.
Training Strategy
PASS employs a two-stage training approach. First, Self-Supervised Pre-training uses a triplet loss to teach the gate to distinguish between similar and different patches without human labels. Second, Task-Oriented Fine-tuning allows the backbone network to adapt to the sparse feature maps generated by the skipping process.
Experimental Results
The researchers evaluated PASS on heavy-duty benchmarks including Human3.6M and MPII for pose estimation, and MOTChallenge for object tracking. Testing on an NVIDIA Jetson Nano revealed substantial performance gains:
- 3D Human Pose Estimation: PASS achieved a 9.43x speedup and reduced Multiply-Accumulate (MAC) operations by 62.61%, while slightly improving accuracy (lower MPJPE).
- Multiple Object Tracking: The system reached a 12.19x speedup and a 73.16% MAC reduction compared to the FairMOT baseline.
- Environmental Robustness: In high-motion scenarios, PASS dynamically adapts its skip strategy, maintaining high tracking accuracy where other sparse models often fail.
Discussion and Conclusion
A key insight of PASS is that even foreground elements can be redundant if their motion is minimal relative to the task. By decoupling acceleration from specific tasks, PASS can be integrated into various backbones like HRNet or ResNet. Furthermore, patch-level skipping leverages block-wise sparse convolution, which is natively supported by modern edge GPU and CPU instructions.
While highly effective, future work could explore multi-scale patch sizes for high-resolution video and further optimize gate overhead in extremely deep networks. PASS demonstrates that the most efficient way to process video is to work smarter by identifying and ignoring redundant information.
Summary updated