Abstract
Temporal Action Detection (TAD) is a critical task for video understanding, yet it remains computationally expensive due to the heavy overhead of feature extractors, which typically account for 95% of total Multiply-Accumulate operations (MACs). Traditional compression methods like channel pruning often result in "slim-and-tall" architectures that fail to leverage GPU parallelism effectively. This paper proposes Progressive Block Drop (PBD), a novel compression strategy that reduces network depth rather than width. By iteratively removing redundant structural blocks and employing cross-depth alignment with parameter-efficient fine-tuning (LoRA), PBD achieves significant efficiency gains. Experimental results across multiple benchmarks, including THUMOS14 and ActivityNet-1.3, show a 25% reduction in computational overhead and a 1.3x inference speedup with virtually no loss in detection accuracy.
Introduction
Temporal Action Detection (TAD) involves identifying the start and end times of specific actions within untrimmed video streams. While foundational for applications like autonomous driving and robotics, the computational demand of TAD pipelines is a major bottleneck. The majority of this cost is concentrated in the feature extractor.
Conventional compression techniques focus on channel pruning to make layers "thinner." However, GPUs are optimized for large matrix multiplications; extremely thin layers often lead to underutilization of parallel processing units. This research shifts the focus toward a "shallow-and-wide" architecture, which maintains hardware efficiency while reducing the total number of operations by dropping entire blocks from the backbone network.
Methods: Progressive Block Drop (PBD)
The proposed PBD method operates through an iterative two-stage cycle designed to transition the model from a deep architecture to a more efficient shallow one.
1. Block Selection
The system evaluates the redundancy of each block within the feature extractor. Blocks where the input and output features exhibit high similarity (low Mean Squared Error) are identified as candidates for removal. The evaluator selects the least impactful block to drop in each iteration.
2. Performance Restoration via Cross-Depth Alignment
To mitigate the accuracy loss typically associated with structural pruning, the authors introduce a Cross-Depth Alignment strategy. This involves using Low-Rank Adaptation (LoRA) to fine-tune the remaining blocks. The objective is to force the compressed model to mimic the intermediate feature representations and final predictions of the original, uncompressed "teacher" model. By dropping blocks progressively rather than all at once, the model can effectively learn to compress the information of multiple original layers into a single remaining block.
Experimental Results
The PBD method was validated on several key benchmarks, including THUMOS14, ActivityNet-1.3, FineAction, and Charades-STA, using various backbones like VideoMAE.
- Efficiency and Speed: On THUMOS14 and ActivityNet, the method achieved a 25% reduction in MACs and a 1.3x inference speedup on standard GPU hardware.
- Accuracy Maintenance: The compression was found to be largely "lossless." On the THUMOS14 dataset, the compressed model actually observed an mAP improvement of 0.94%, likely due to the regularization effect of removing redundant parameters.
- Orthogonality: PBD is complementary to existing techniques. When combined with traditional channel pruning, it provides even greater efficiency gains than either method alone.
Discussion and Conclusion
This work highlights the significant depth redundancy present in modern video backbone networks. By prioritizing a "shallow-and-wide" design, PBD aligns model architecture with the parallel processing strengths of modern hardware. While the progressive training cycle requires more initial training time than simple fine-tuning, the resulting models offer superior deployment characteristics for real-time video applications. Future research may explore automated stopping criteria for the drop cycle and the application of this logic to multi-modal models incorporating audio and text features.
Summary updated