DPCNet: Dual Path Multi-Excitation Collaborative Network for Facial Expression Representation Learning in Videos

Yan Wang, Yixuan Sun, Wei Song, Shuyong Gao, Yiwen Huang, Zhaoyu Chen, Weifeng Ge, Wenqiang Zhang

Abstract

Facial expression learning in videos is challenging since it is concerned with spatial-channel feature representation and temporal relationships while consuming significant computation resources. To mitigate this issue, we propose a Dual Path multi-excitation Collaborative Network (DPCNet) to learn the critical information for facial expression representation from fewer keyframes in videos. Specifically, the DPCNet learns the important regions and keyframes from a pair of four view-grouped frames by multi-excitation modules and produces dual-path representations of one video with consistency under two regularization strategies. A spatial-frame excitation module (SFEM) and a channel-temporal aggregation module (CTAM) are successively introduced to learn spatial-frame representation and generate complementary channel-temporal aggregation, respectively. Moreover, we design a multi-frame regularization loss based on self-supervised learning to enforce the representation of multiple frames in the dual view to be semantically coherent. To obtain consistent prediction probabilities from the dual path, we further propose a dual path regularization loss, aiming to minimize the divergence between the distributions of two-path embeddings. The performance of the DPCNet is evaluated on benchmark datasets (i.e., CK+, AFEW, and DFEW). Extensive experiments and ablation studies show that our proposed DPCNet can vastly improve the performance of video-based FER and achieve state-of-the-art results on the large-scale DFEW dataset. Code is available.
BibTeX
@inproceedings{wang2022dpcnet,
  author = {Yan Wang and Yixuan Sun and Wei Song and Shuyong Gao and Yiwen Huang and Zhaoyu Chen and Weifeng Ge and Wenqiang Zhang},
  title = {{DPCNet: Dual Path Multi-Excitation Collaborative Network for Facial Expression Representation Learning in Videos}},
  booktitle = {ACM Multimedia},
  year = {2022},
  doi = {10.1145/3503161.3547865},
}

Loading related content...

Figure