Temporal Action Localization with Cross Layer Task Decoupling and Refinement

Qiang Li, Di Liu, Jun Kong, Sen Li, Hui Xu, Jianzhong WangAAAI 2025 · DOI

Abstract

Temporal Action Localization (TAL) requires simultaneous action classification and boundary regression, tasks that often demand conflicting feature representations. We present CLTDR-GMG, a framework that addresses this conflict through Cross-Layer Task Decoupling and Refinement (CLTDR) and an efficient Gated Multi-Granularity (GMG) encoder. By leveraging semantic-rich features from higher pyramid layers for classification and boundary-sharp features from lower layers for localization, our model achieves state-of-the-art performance on five major benchmarks. Notably, our Fourier-based global context module significantly reduces parameter counts compared to Transformer-based alternatives while maintaining high accuracy.

Introduction

Temporal Action Localization (TAL) involves identifying both the category and the temporal boundaries of actions in untrimmed video sequences. Current research faces two primary bottlenecks:

  1. The Feature Conflict: Classification benefits from coarse, high-level semantic information, whereas localization requires fine-grained, frame-level details. Standard models often force these dual tasks to share identical input features, leading to sub-optimal performance.
  2. The Global Context Tax: Capturing long-range dependencies is crucial for understanding action context. While Self-Attention mechanisms in Transformers are effective, they are computationally expensive and memory-intensive for long videos.

Methodology

Gated Multi-Granularity (GMG) Module

To capture temporal dependencies efficiently, we introduce the GMG module. Unlike heavy Self-Attention, the GMG module processes features across three granularities:

  • Instant: Frame-level features via fully-connected layers.
  • Local: Short-term context via 1D depth-wise convolutions.
  • Global: Long-range dependencies using a 1D Fast Fourier Transform (FFT). Operating in the frequency domain allows the model to capture global patterns with approximately 2.7x fewer parameters than standard attention mechanisms.

A learned gating mechanism selectively fuses these branches to optimize feature representation while reducing redundancy.

The Gated Multi-Granularity Module (GMG) utilizes FFT for efficient global context alongside local and instant branches.

Cross Layer Task Decoupling and Refinement (CLTDR)

The CLTDR decoder explicitly separates the feature needs of classification and regression by referencing adjacent layers in the feature pyramid:

  • Decoupled Classification Module (DCM): For a given layer $l$, the DCM pulls stronger semantic context from the higher layer $l+1$.
  • Decoupled Regression Module (DRM): To refine boundaries, the DRM retrieves finer temporal details from the lower layer $l-1$.
  • RefineHead: A final fusion module harmonizes classification and localization outputs, ensuring that high-confidence predictions are paired with accurate temporal boundaries.
The CLTDR architecture decouples tasks by pulling features from adjacent pyramid layers (DCM and DRM) before final refinement.

Experimental Results

CLTDR-GMG was evaluated on five challenging benchmarks, including THUMOS14, MultiTHUMOS, EPIC-KITCHENS-100, ActivityNet-1.3, and HACS. Key findings include:

  • State-of-the-Art Accuracy: On THUMOS14, the model achieved an average mAP of 74.3%, outperforming established models like TriDet.
  • Significant Gains in Dense TAL: On MultiTHUMOS, the model reached 44.9% average mAP, a substantial improvement over the previous state-of-the-art (~37.5%).
  • Efficiency: The FFT-based global extractor reduced the parameter count from 26.1M to 9.5M compared to Self-Attention baselines without sacrificing performance.

Conclusion

CLTDR-GMG demonstrates that task decoupling at the feature level is essential for high-performance Temporal Action Localization. By providing each task with its ideal "zoom level" through cross-layer feature access and replacing expensive attention with efficient Fourier transforms, we provide a scalable and accurate solution for video understanding. Future work will focus on extending this decoupling logic to the extreme layers of the feature pyramid and optimizing throughput for real-time applications.

Summary updated

BibTeX
@inproceedings{li2025temporal,
  author = {Qiang Li and Di Liu and Jun Kong and Sen Li and Hui Xu and Jianzhong Wang},
  title = {{Temporal Action Localization with Cross Layer Task Decoupling and Refinement}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i5.32516},
}
Figure