Stacking-Based Attention Temporal Convolutional Network for Action Segmentation

Liu Yang, Yu Jiang, Junkun Hong, Zhenjie Wu, Zhan Yang, Jun LongICASSP 2023 · OpenReview

Abstract

Action segmentation plays an important role in video understanding, which is implemented by frame-wise action classification. Recent works on action segmentation capture long-term dependencies by increasing temporal convolution layers in Temporal Convolution Networks (TCNs). However, high layers in TCNs are more coarse access to video features, resulting in the loss of fine-grained information for frame-wise action classification. To address the above issues, we propose a novel Attention-based Temporal Convolution (ATC) block to capture fine-grained information of temporal dependencies for frame-wise action classification by self-attention mechanism. Via stacking ATC blocks, we design a Stacking-based Attention Temporal Convolutional Network (SATC) to adaptively capture long-term and short-term dependencies, according to the semantic similarity of features on different temporal receptive fields simultaneously. The experimental results demonstrate that our SATC outperforms other baselines on all three challenging datasets: GTEA, 50Salads and Breakfast.
BibTeX
@inproceedings{yang2023stacking,
  author = {Liu Yang and Yu Jiang and Junkun Hong and Zhenjie Wu and Zhan Yang and Jun Long},
  title = {{Stacking-Based Attention Temporal Convolutional Network for Action Segmentation}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2023},
  url = {https://openreview.net/forum?id=-u1OAG2BIV4},
}

Loading related content...

Figure