Deep Concept-wise Temporal Convolutional Networks for Action Localization

Xin Li, Tianwei Lin, Xiao Liu, Wangmeng Zuo, Chao Li, Xiang Long, Dongliang He, Fu Li, Shilei Wen, Chuang Gan

TL;DR

Researchers have discovered that simply making 1D temporal convolutional networks (TCNs) deeper actually hurts their ability to classify actions in video. To fix this, they developed Concept-wise Temporal Convolution (CTC), a novel layer that processes feature channels independently with shared weights, enabling much deeper architectures (C-TCN) that achieved a massive 21.7% relative improvement on the THUMOS’14 benchmark.

What Problem Does This Paper Address?

Temporal Action Localization (TAL) is the video equivalent of object detection: given an untrimmed video, the system must identify exactly when an action starts and ends, and what that action is.

In the world of 2D images, "deeper is better" is a golden rule—stacking more layers in a CNN generally improves performance. However, the authors found a strange phenomenon in 1D temporal networks: as you add more conventional convolutional layers, the network's ability to classify the action (e.g., "Is this a high jump or a long jump?") actually degrades.

The authors hypothesize that standard convolutions mix all feature channels (which they call "latent concepts") too aggressively. By the time the data reaches the final layers of a deep network, these concepts are so "excessively recombined" that they lose their discriminative power, making it harder for the model to tell different actions apart.

Methods

The core innovation is the Concept-wise Temporal Convolution (CTC) layer. Here is how it differs from standard approaches:

  1. Standard Convolution: Every filter looks at all input channels simultaneously, mixing them together.
  2. Group Convolution: Channels are split into independent groups, but each group uses its own unique parameters.
  3. Concept-wise Convolution (The New Way): The layer treats each channel as a separate "concept." It applies temporal filters to each channel individually. Crucially, it shares the same filter parameters across all channels.

By sharing weights, the model learns universal temporal patterns (like "gradual increase" or "sudden peak") that apply to many different semantic concepts. This prevents the "concept mixing" problem while significantly enriching the representation.

The authors built a deep network called C-TCN using these layers. It utilizes a Feature Pyramid Network (FPN) backbone to handle actions of varying lengths—from a quick "basketball dunk" to a long "billiards" match—and employs two "heads" for the final output: one for classification and one for boundary regression (fine-tuning the start/end times).

Datasets

The model was evaluated on two major industry-standard benchmarks:

  • THUMOS’14: A challenging dataset with 20 action classes. It is known for having many short action instances packed into untrimmed videos.
  • ActivityNet-1.3: A large-scale dataset containing 200 different daily activities (like "walking the dog" or "vacuuming") across 20,000 videos.

Key Insights and Ideas

  • Weight Sharing is Essential: The ablation studies revealed that simply using group convolutions (to keep channels separate) isn't enough. The performance boost specifically comes from sharing the weights across those groups, which forces the network to learn more robust, generalized temporal primitives.
  • Overcoming the "Depth Barrier": While standard TCNs saw performance drop after just 15 layers, the C-TCN continued to improve as it scaled up to 60 layers.
  • Temporal Data Augmentation: The authors introduced "temporal random move" and "random crop" techniques. They found that for deep 1D networks, these augmentations are critical to prevent overfitting, a step often overlooked in previous TAL research.

Key Results

  • State-of-the-Art on THUMOS’14: C-TCN increased the mean Average Precision (mAP) at a 0.5 threshold from the previous best of 42.8% to 52.1%.
  • ActivityNet Performance: It achieved an average mAP of 31.1%, outperforming other high-profile models like BSN and TAL-Net.
  • Consistency: Across all tested depths (15, 30, 45, and 60 layers), the C-TCN consistently outperformed standard TCNs, proving that the architecture is fundamentally more stable for deep temporal modeling.

Limitations and Future Work

Despite the accuracy gains, the qualitative analysis showed that the model still struggles with fine-grained classification in complex scenes. For example, it correctly localized a "Frisbee Catch" but misclassified it as a "Soccer Penalty" because the temporal motions were similar.

The authors suggest that future work could involve:

  • Applying C-TCN architectures to general video classification tasks beyond just localization.
  • Integrating more sophisticated video-level classification results to help resolve "look-alike" action errors.
  • Exploring the computational efficiency of these deeper 1D networks for real-time applications.

Summary updated

Abstract

Existing action localization approaches adopt shallow temporal convolutional networks (i.e., TCN) on 1D feature map extracted from video frames. In this paper, we empirically find that stacking more conventional temporal convolution layers actually deteriorates action classification performance, possibly ascribing to that all channels of 1D feature map, which generally are highly abstract and can be regarded as latent concepts, are excessively recombined in temporal convolution. To address this issue, we introduce a novel concept-wise temporal convolutional network (C-TCN) as an alternative to TCN for training deeper action localization networks. To address this issue, we introduce a novel concept-wise temporal convolution (CTC) layer as an alternative to conventional temporal convolution layer for training deeper action localization networks. Instead of recombining latent concepts, CTC layer deploys a number of temporal filters to each concept separately with shared filter parameters across concepts. Thus can capture common temporal patterns of different concepts and significantly enrich representation ability. Via stacking CTC layers, we proposed a deep concept-wise temporal convolutional network (C-TCN), which boosts the state-of-the-art action localization performance on THUMOS'14 from 42.8 to 52.1 in terms of mAP(%), achieving a relative improvement of 21.7%. Favorable result is also obtained on ActivityNet.
BibTeX
@inproceedings{li2020deep,
  author = {Xin Li and Tianwei Lin and Xiao Liu and Wangmeng Zuo and Chao Li and Xiang Long and Dongliang He and Fu Li and Shilei Wen and Chuang Gan},
  title = {{Deep Concept-wise Temporal Convolutional Networks for Action Localization}},
  booktitle = {ACM Multimedia},
  year = {2020},
  doi = {10.1145/3394171.3413860},
}

Loading related content...

Figure