Temporal Action Detection Model Compression by Progressive Block Drop

Xiaoyong Chen, Yong Guo, Jiaming Liang, Sitong Zhuang, Runhao Zeng, Xiping Hu

TL;DR

This paper introduces Progressive Block Drop, a novel model compression method for Temporal Action Detection (TAD) that reduces network depth rather than width. By iteratively removing redundant blocks and applying a parameter-efficient cross-depth alignment technique, the approach achieves a 25% reduction in computational overhead and a 1.29× hardware speedup with zero loss in accuracy. Crucially, keeping the network layers wide preserves GPU parallelization efficiency far better than traditional channel pruning.

What Problem Does This Paper Address?

Temporal Action Detection (TAD) involves localizing the start and end times of actions within long, untrimmed videos and classifying them. As feature extractors grow larger to handle complex video data, their computational demands skyrocket, making them difficult to deploy in resource-constrained environments like autonomous driving and robotics.

Interestingly, the feature extraction phase accounts for a staggering 95% of a TAD model's total computation because it processes many video segments individually, whereas the detection head only processes a single video-level feature.

Figure 1. Comparisons between reducing depth and reducing width. For a 768-frame video sequence with a resolution of 1602, using VideoMAE-S as the feature extractor and Actionformer as the detection head, 95% of the computational overhead comes from the feature extractor. Most pruning methods reduce the size of layer weight matrices, leading to a slim-and-tall network structure. In contrast, our progressive block drop method reduces network depth, achieving 1.19× faster inference at the same computational cost. These findings, detailed in Section 5.6, suggest that our approach results in a more hardware-efficient model.

To compress these models, researchers typically turn to channel pruning, which slims down the hidden layer dimensions. However, this creates a "slim-and-tall" network architecture. For modern GPUs, executing operations on multiple small matrices is highly inefficient compared to processing larger matrices due to reduced parallelization. The authors set out to solve this by compressing the model from a depth perspective, creating a hardware-friendly "shallow-and-wide" architecture.

Methods

The authors propose a multi-step Progressive Block Drop framework that iteratively shrinks the feature extractor block-by-block. Each iteration consists of two main stages:

  1. Block Selection and Removal: The framework evaluates the importance of each block by creating sub-networks that drop one block at a time. A block selection evaluator measures the performance of these sub-networks on the training set using the mean Average Precision (mAP) metric, automatically removing the least critical block.
  2. Performance Restoration via Cross-Depth Alignment: Dropping a block alters the feature flow to subsequent layers. To fix this structural gap without expensive full-parameter fine-tuning, the authors integrate Low-Rank Adaptation (LoRA) into the remaining attention modules. They then apply a Cross-Depth Alignment strategy (a form of knowledge distillation) that forces the pruned model's intermediate features to match the uncompressed model's features.

Additionally, prediction-level alignment is used, employing Kullback-Leibler (KL) divergence for action classification and Generalized Intersection over Union (GIoU) loss for bounding box regression to ensure the compressed model retains high detection accuracy.

Figure 3. The diagram of our progressive block drop method. Our approach adopts a multi-step progressive compression strategy. At each iteration 1) we evaluate the importance of each block and drop the least important block, and 2) we use parameter-efficient tuning techniques, and recover performance by learning from the uncompressed model through feature-level and prediction-level alignment.

Datasets

The method was evaluated on several prominent video understanding benchmarks:

  • THUMOS14: A challenging dataset of short videos with dense action annotations (averaging over 15 action instances per video).
  • ActivityNet-1.3: A large-scale benchmark containing approximately 20,000 long, untrimmed videos.
  • FineAction: A newer benchmark focused on fine-grained temporal action localization.
  • Charades-STA: Used to validate the generalizability of the method on a natural language action localization (grounding) task.

Key Insights and Ideas

  • Severe Depth Redundancy: By analyzing the Mean Squared Error (MSE) between the inputs and outputs of individual blocks, the authors discovered that several blocks have an MSE close to 0. This means they perform almost identity mappings and are functionally redundant.
  • Shallow-and-Wide > Deep-and-Thin: For the exact same number of Multiply-Accumulate operations (MACs), a shallower network with wider layers runs faster on a GPU than a deeper, narrower network because it maximizes matrix-multiplication parallelism.
  • The Progressive Necessity: Dropping multiple blocks simultaneously causes a severe, unrecoverable drop in accuracy. Dropping them progressively allows the remaining layers to gradually absorb and compensate for the missing information.
Figure 2. Analysis of TAD models at block level on THUMOS14 datasets, using VideoMAE-S. Blue curve: impact of dropping a single block on the model’s detection accuracy, where most cases result in minimal accuracy degradation. Green line: MSE of the input and output features for each block. Some blocks exhibit MSE values close to 0, indicating that these blocks are redundant.

Key Results

  • Lossless Compression & Speedup: On THUMOS14 and ActivityNet-1.3, the method cut computational costs (MACs) by ~25% while actually improving performance over the uncompressed baseline (e.g., +0.94% mAP on THUMOS14 with a VideoMAE-S backbone).
  • Superior Hardware Efficiency: When compared directly to channel pruning at an identical MAC count, Progressive Block Drop achieved a 1.29× speedup versus pruning's 1.09× speedup, proving its hardware-friendly nature.
  • Orthogonality to Width Pruning: The method can be stacked with traditional channel pruning. Combining both approaches yielded a 30% reduction in MACs and an additional 1.3× acceleration without sacrificing accuracy.
Figure 4. Comparison and compatibility with the pruning method. Bubble size represents the model’s computational complexity. Combining our method with pruning enables further 1.3× accel- eration, demonstrating our compatibility with pruning.

Limitations and Future Work

A detailed error analysis using the DETAD tool revealed that while Progressive Block Drop significantly improves localization performance (reducing background and localization errors), it comes at a slight cost to classification accuracy (with classification error rising slightly from 0.7 to 1.1).

Future work could explore more sophisticated alignment loss functions to better balance classification and localization retention, or extend this depth-wise compression paradigm to multi-modal video-language models.

Summary updated

Abstract

Temporal action detection (TAD) aims to identify and localize action instances in untrimmed videos, which is essential for various video understanding tasks. However, recent improvements in model performance, driven by larger feature extractors and datasets, have led to increased computational demands. This presents a challenge for applications like autonomous driving and robotics, which rely on limited computational resources. While existing channel pruning methods can compress these models, reducing the number of channels often hinders the parallelization efficiency of GPU, due to the inefficient multiplication between small matrices. Instead of pruning channels, we propose a Progressive Block Drop method that reduces model depth while retaining layer width. In this way, we still use large matrices for computation but reduce the number of multiplications. Our approach iteratively removes redundant blocks in two steps: first, we drop blocks with minimal impact on model performance; and second, we employ a parameter-efficient cross-depth alignment technique, fine-tuning the pruned model to restore model accuracy. Our method achieves a 25% reduction in computational overhead on two TAD benchmarks (THUMOS14 and ActivityNet-1.3) to achieve lossless compression. More critically, we empirically show that our method is orthogonal to channel pruning methods and can be combined with it to yield further efficiency gains.
BibTeX
@inproceedings{chen2025temporal,
  author = {Xiaoyong Chen and Yong Guo and Jiaming Liang and Sitong Zhuang and Runhao Zeng and Xiping Hu},
  title = {{Temporal Action Detection Model Compression by Progressive Block Drop}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2025},
  doi = {10.1109/cvpr52734.2025.02721},
}
Figure