TL;DR
This paper introduces Progressive Block Drop, a novel model compression method for Temporal Action Detection (TAD) that reduces network depth rather than width. By iteratively removing redundant blocks and applying a parameter-efficient cross-depth alignment technique, the approach achieves a 25% reduction in computational overhead and a 1.29× hardware speedup with zero loss in accuracy. Crucially, keeping the network layers wide preserves GPU parallelization efficiency far better than traditional channel pruning.
What Problem Does This Paper Address?
Temporal Action Detection (TAD) involves localizing the start and end times of actions within long, untrimmed videos and classifying them. As feature extractors grow larger to handle complex video data, their computational demands skyrocket, making them difficult to deploy in resource-constrained environments like autonomous driving and robotics.
Interestingly, the feature extraction phase accounts for a staggering 95% of a TAD model's total computation because it processes many video segments individually, whereas the detection head only processes a single video-level feature.
To compress these models, researchers typically turn to channel pruning, which slims down the hidden layer dimensions. However, this creates a "slim-and-tall" network architecture. For modern GPUs, executing operations on multiple small matrices is highly inefficient compared to processing larger matrices due to reduced parallelization. The authors set out to solve this by compressing the model from a depth perspective, creating a hardware-friendly "shallow-and-wide" architecture.
Methods
The authors propose a multi-step Progressive Block Drop framework that iteratively shrinks the feature extractor block-by-block. Each iteration consists of two main stages:
- Block Selection and Removal: The framework evaluates the importance of each block by creating sub-networks that drop one block at a time. A block selection evaluator measures the performance of these sub-networks on the training set using the mean Average Precision (mAP) metric, automatically removing the least critical block.
- Performance Restoration via Cross-Depth Alignment: Dropping a block alters the feature flow to subsequent layers. To fix this structural gap without expensive full-parameter fine-tuning, the authors integrate Low-Rank Adaptation (LoRA) into the remaining attention modules. They then apply a Cross-Depth Alignment strategy (a form of knowledge distillation) that forces the pruned model's intermediate features to match the uncompressed model's features.
Additionally, prediction-level alignment is used, employing Kullback-Leibler (KL) divergence for action classification and Generalized Intersection over Union (GIoU) loss for bounding box regression to ensure the compressed model retains high detection accuracy.
Datasets
The method was evaluated on several prominent video understanding benchmarks:
- THUMOS14: A challenging dataset of short videos with dense action annotations (averaging over 15 action instances per video).
- ActivityNet-1.3: A large-scale benchmark containing approximately 20,000 long, untrimmed videos.
- FineAction: A newer benchmark focused on fine-grained temporal action localization.
- Charades-STA: Used to validate the generalizability of the method on a natural language action localization (grounding) task.
Key Insights and Ideas
- Severe Depth Redundancy: By analyzing the Mean Squared Error (MSE) between the inputs and outputs of individual blocks, the authors discovered that several blocks have an MSE close to 0. This means they perform almost identity mappings and are functionally redundant.
- Shallow-and-Wide > Deep-and-Thin: For the exact same number of Multiply-Accumulate operations (MACs), a shallower network with wider layers runs faster on a GPU than a deeper, narrower network because it maximizes matrix-multiplication parallelism.
- The Progressive Necessity: Dropping multiple blocks simultaneously causes a severe, unrecoverable drop in accuracy. Dropping them progressively allows the remaining layers to gradually absorb and compensate for the missing information.
Key Results
- Lossless Compression & Speedup: On THUMOS14 and ActivityNet-1.3, the method cut computational costs (MACs) by ~25% while actually improving performance over the uncompressed baseline (e.g., +0.94% mAP on THUMOS14 with a VideoMAE-S backbone).
- Superior Hardware Efficiency: When compared directly to channel pruning at an identical MAC count, Progressive Block Drop achieved a 1.29× speedup versus pruning's 1.09× speedup, proving its hardware-friendly nature.
- Orthogonality to Width Pruning: The method can be stacked with traditional channel pruning. Combining both approaches yielded a 30% reduction in MACs and an additional 1.3× acceleration without sacrificing accuracy.
Limitations and Future Work
A detailed error analysis using the DETAD tool revealed that while Progressive Block Drop significantly improves localization performance (reducing background and localization errors), it comes at a slight cost to classification accuracy (with classification error rising slightly from 0.7 to 1.1).
Future work could explore more sophisticated alignment loss functions to better balance classification and localization retention, or extend this depth-wise compression paradigm to multi-modal video-language models.
Summary updated