Temporal Action Detection Model Compression by Progressive Block Drop

Xiaoyong Chen, Yong Guo, Jiaming Liang, Sitong Zhuang, Runhao Zeng, Xiping Hu

Abstract

Temporal Action Detection (TAD) is a critical task for video understanding, yet it remains computationally expensive due to the heavy overhead of feature extractors, which typically account for 95% of total Multiply-Accumulate operations (MACs). Traditional compression methods like channel pruning often result in "slim-and-tall" architectures that fail to leverage GPU parallelism effectively. This paper proposes Progressive Block Drop (PBD), a novel compression strategy that reduces network depth rather than width. By iteratively removing redundant structural blocks and employing cross-depth alignment with parameter-efficient fine-tuning (LoRA), PBD achieves significant efficiency gains. Experimental results across multiple benchmarks, including THUMOS14 and ActivityNet-1.3, show a 25% reduction in computational overhead and a 1.3x inference speedup with virtually no loss in detection accuracy.

Introduction

Temporal Action Detection (TAD) involves identifying the start and end times of specific actions within untrimmed video streams. While foundational for applications like autonomous driving and robotics, the computational demand of TAD pipelines is a major bottleneck. The majority of this cost is concentrated in the feature extractor.

Conventional compression techniques focus on channel pruning to make layers "thinner." However, GPUs are optimized for large matrix multiplications; extremely thin layers often lead to underutilization of parallel processing units. This research shifts the focus toward a "shallow-and-wide" architecture, which maintains hardware efficiency while reducing the total number of operations by dropping entire blocks from the backbone network.

Methods: Progressive Block Drop (PBD)

The proposed PBD method operates through an iterative two-stage cycle designed to transition the model from a deep architecture to a more efficient shallow one.

1. Block Selection

The system evaluates the redundancy of each block within the feature extractor. Blocks where the input and output features exhibit high similarity (low Mean Squared Error) are identified as candidates for removal. The evaluator selects the least impactful block to drop in each iteration.

2. Performance Restoration via Cross-Depth Alignment

To mitigate the accuracy loss typically associated with structural pruning, the authors introduce a Cross-Depth Alignment strategy. This involves using Low-Rank Adaptation (LoRA) to fine-tune the remaining blocks. The objective is to force the compressed model to mimic the intermediate feature representations and final predictions of the original, uncompressed "teacher" model. By dropping blocks progressively rather than all at once, the model can effectively learn to compress the information of multiple original layers into a single remaining block.

Overview of the Progressive Block Drop (PBD) cycle and Cross-Depth Alignment strategy.

Experimental Results

The PBD method was validated on several key benchmarks, including THUMOS14, ActivityNet-1.3, FineAction, and Charades-STA, using various backbones like VideoMAE.

  • Efficiency and Speed: On THUMOS14 and ActivityNet, the method achieved a 25% reduction in MACs and a 1.3x inference speedup on standard GPU hardware.
  • Accuracy Maintenance: The compression was found to be largely "lossless." On the THUMOS14 dataset, the compressed model actually observed an mAP improvement of 0.94%, likely due to the regularization effect of removing redundant parameters.
  • Orthogonality: PBD is complementary to existing techniques. When combined with traditional channel pruning, it provides even greater efficiency gains than either method alone.
Performance vs. computational cost (MACs) comparison between baseline, pruning, and the proposed PBD method.

Discussion and Conclusion

This work highlights the significant depth redundancy present in modern video backbone networks. By prioritizing a "shallow-and-wide" design, PBD aligns model architecture with the parallel processing strengths of modern hardware. While the progressive training cycle requires more initial training time than simple fine-tuning, the resulting models offer superior deployment characteristics for real-time video applications. Future research may explore automated stopping criteria for the drop cycle and the application of this logic to multi-modal models incorporating audio and text features.

Summary updated

Abstract

Temporal action detection (TAD) aims to identify and localize action instances in untrimmed videos, which is essential for various video understanding tasks. However, recent improvements in model performance, driven by larger feature extractors and datasets, have led to increased computational demands. This presents a challenge for applications like autonomous driving and robotics, which rely on limited computational resources. While existing channel pruning methods can compress these models, reducing the number of channels often hinders the parallelization efficiency of GPU, due to the inefficient multiplication between small matrices. Instead of pruning channels, we propose a **Progressive Block Drop** method that reduces model depth while retaining layer width. In this way, we still use large matrices for computation but reduce the number of multiplications. Our approach iteratively removes redundant blocks in two steps: first, we drop blocks with minimal impact on model performance; and second, we employ a parameter-efficient cross-depth alignment technique, fine-tuning the pruned model to restore model accuracy. Our method achieves a 25% reduction in computational overhead on two TAD benchmarks (THUMOS14 and ActivityNet-1.3) to achieve lossless compression. More critically, we empirically show that our method is orthogonal to channel pruning methods and can be combined with it to yield further efficiency gains.
BibTeX
@inproceedings{chen2025temporal,
  author = {Xiaoyong Chen and Yong Guo and Jiaming Liang and Sitong Zhuang and Runhao Zeng and Xiping Hu},
  title = {{Temporal Action Detection Model Compression by Progressive Block Drop}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2025},
  doi = {10.1109/cvpr52734.2025.02721},
}
Figure