TwinFormer: Fine-to-Coarse Temporal Modeling for Long-Term Action Recognition
Lists
Only you see this
Abstract
The long-term action in untrimmed video generally contains multiple sub-actions, among which various semantic patterns exist ( <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">e.g.</i> , the co-occurrence or sequentiality between sub-actions). These semantic patterns are temporally coarse, and correlated with multiple local contexts which encode the local temporal evolution of visual elements ( <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">e.g.</i> , hands, objects) in videos. The local contexts and semantic patterns form the inherent fine-to-coarse temporal structure of long-term actions, which is neglected by existing works. Accordingly, in this work we propose TwinFormer, which exploits a novel fine-to-coarse temporal modeling manner to uncover the temporal structure of long-term actions. The proposed TwinFormer consists of a pair of twin encoders with the same structural design, namely Localcontext Encoder and Semantic-pattern Encoder, and a Temporalbridged Attention to bridge the two twin encoders. The Localcontext Encoder aims to model the local contexts in the longterm action. And the Temporal-bridged Attention is designed to correlate the local contexts with semantic patterns. Furthermore, the Semantic-pattern Encoder reveals the temporal evolution of semantic patterns. Experimental results on three benchmarks demonstrate the effectiveness of the proposed model.
BibTeX
@article{zhou2024twinformer,
author = {Jiaming Zhou and Kun-Yu Lin and Yu-Kun Qiu and Wei‐Shi Zheng},
title = {{TwinFormer: Fine-to-Coarse Temporal Modeling for Long-Term Action Recognition}},
journal = {IEEE Trans. Multimedia},
year = {2024},
url = {https://openreview.net/forum?id=CRw5k5x8y9},
}Loading related content...