Deep Concept-wise Temporal Convolutional Networks for Action Localization

Xin Li, Tianwei Lin, Xiao Liu, Chuang Gan, Wangmeng Zuo, Chao Li, Xiang Long, Dongliang He, Fu Li, Shilei Wen

Abstract

Existing action localization approaches adopt shallow temporal convolutional networks (TCN) on 1D feature maps extracted from video frames. In this paper, we empirically find that stacking more conventional temporal convolution layers actually deteriorates action classification performance, possibly ascribing to that all channels of 1D feature map, which generally are highly abstract and can be regarded as latent concepts, are excessively recombined in temporal convolution. To address this issue, we introduce a novel concept-wise temporal convolution (CTC) layer as an alternative to conventional temporal convolution layer for training deeper action localization networks. Instead of recombining latent concepts, CTC layer deploys a number of temporal filters to each concept separately with shared filter parameters across concepts. Thus can capture common temporal patterns of different concepts and significantly enrich representation ability. Via stacking CTC layers, we proposed a deep concept-wise temporal convolutional network (C-TCN), which boosts the state-of-the-art action localization performance on THUMOS'14 from 42.8 to 52.1 in terms of mAP(%), achieving a relative improvement of 21.7%. Favorable result is also obtained on ActivityNet.

Introduction

Temporal Action Localization (TAL) is a critical task in video understanding, requiring the identification of both the temporal boundaries (start and end times) and the semantic categories of actions within untrimmed videos. While deep convolutional neural networks (CNNs) have revolutionized 2D image tasks by leveraging increased depth, a similar trend has been missing in 1D temporal networks for TAL.

Researchers observed that simply increasing the depth of standard Temporal Convolutional Networks (TCNs) often leads to a performance plateau or even degradation, particularly in action classification. This "depth barrier" is hypothesized to stem from the "excessive recombination" of feature channels—referred to here as "latent concepts"—in standard 1D convolutions. As data passes through many layers, these concepts are mixed so aggressively that they lose the discriminative power necessary for fine-grained action classification.

Methods: Concept-wise Temporal Convolution

To overcome the limitations of standard TCNs, this work introduces the Concept-wise Temporal Convolution (CTC) layer. Unlike standard convolutions that mix all input channels in every filter, the CTC layer treats each channel as an independent semantic concept.

Comparison of Standard, Group, and Concept-wise Temporal Convolutions.

As shown in the figure above, the CTC layer differs from existing approaches in several ways:

  1. Standard Convolution: Every filter looks at all input channels simultaneously, mixing them together and leading to high recombination.
  2. Group Convolution: Channels are split into independent groups, but each group uses its own unique parameters.
  3. Concept-wise Convolution: The layer applies temporal filters to each channel individually. Crucially, it shares the same filter parameters across all channels.

By sharing weights, the model learns universal temporal patterns (like "gradual increase" or "sudden peak") that apply to many different semantic concepts. This prevents the "concept mixing" problem while significantly enriching the representation. The resulting C-TCN architecture utilizes these layers within a Feature Pyramid Network (FPN) backbone to handle actions of varying lengths.

Key Insights and Results

The primary advantage of the C-TCN is its stability and performance at greater depths. While standard TCNs see classification loss increase as the network grows from 15 to 60 layers, the C-TCN maintains low loss and continues to improve.

Localization and classification loss vs. network depth for TCN and C-TCN.

Performance Benchmarks

The model was evaluated on two major industry-standard benchmarks:

  • THUMOS’14: C-TCN increased the mean Average Precision (mAP) at a 0.5 threshold from the previous best of 42.8% to 52.1%, a 21.7% relative improvement.
  • ActivityNet-1.3: It achieved an average mAP of 31.1%, outperforming other high-profile models like BSN and TAL-Net.

Ablation and Augmentation

Ablation studies revealed that weight sharing is essential; simply using group convolutions to keep channels separate is not enough. The performance boost specifically comes from sharing weights across those groups, forcing the network to learn more robust, generalized temporal primitives. Additionally, the authors introduced "temporal random move" and "random crop" techniques, finding that for deep 1D networks, these augmentations are critical to prevent overfitting.

Conclusion

The C-TCN architecture successfully breaks the depth barrier in temporal action localization. By deploying concept-wise convolutions with shared weights, the model can scale up to 60 layers while maintaining high discriminative power. While the model still faces challenges with fine-grained classification in complex scenes where temporal motions are similar, it represents a significant step forward in deep temporal modeling for video analysis.

Summary updated

Abstract

Existing action localization approaches adopt shallow temporal convolutional networks (\ie, TCN) on 1D feature map extracted from video frames. In this paper, we empirically find that stacking more conventional temporal convolution layers actually deteriorates action classification performance, possibly ascribing to that all channels of 1D feature map, which generally are highly abstract and can be regarded as latent concepts, are excessively recombined in temporal convolution. To address this issue, we introduce a novel concept-wise temporal convolution (CTC) layer as an alternative to conventional temporal convolution layer for training deeper action localization networks. Instead of recombining latent concepts, CTC layer deploys a number of temporal filters to each concept separately with shared filter parameters across concepts. Thus can capture common temporal patterns of different concepts and significantly enrich representation ability. Via stacking CTC layers, we proposed a deep concept-wise temporal convolutional network (C-TCN), which boosts the state-of-the-art action localization performance on THUMOS'14 from 42.8 to 52.1 in terms of mAP(\%), achieving a relative improvement of 21.7\%. Favorable result is also obtained on ActivityNet.
BibTeX
@inproceedings{li2020deep,
  author = {Xin Li and Tianwei Lin and Xiao Liu and Chuang Gan and Wangmeng Zuo and Chao Li and Xiang Long and Dongliang He and Fu Li and Shilei Wen},
  title = {{Deep Concept-wise Temporal Convolutional Networks for Action Localization}},
  booktitle = {ACM Multimedia},
  year = {2020},
  doi = {10.1145/3394171.3413860},
}
Figure