DPCNet: Dual Path Multi-Excitation Collaborative Network for Facial Expression Representation Learning in Videos

Yan Wang

Abstract

Facial expression learning in videos is challenging since it is concerned with spatial-channel feature representation and temporal relationships while consuming significant computation resources. To mitigate this issue, we propose a Dual Path multi-excitation Collaborative Network (DPCNet) to learn the critical information for facial expression representation from fewer keyframes in videos. Specifically, the DPCNet learns the important regions and keyframes from a pair of four view-grouped frames by multi-excitation modules and produces dual-path representations of one video with consistency under two regularization strategies. A spatial-frame excitation module (SFEM) and a channel-temporal aggregation module (CTAM) are successively introduced to learn spatial-frame representation and generate complementary channel-temporal aggregation, respectively. Moreover, we design a multi-frame regularization loss based on self-supervised learning to enforce the representation of multiple frames in the dual view to be semantically coherent. To obtain consistent prediction probabilities from the dual path, we further propose a dual path regularization loss, aiming to minimize the divergence between the distributions of two-path embeddings. The performance of the DPCNet is evaluated on benchmark datasets (i.e., CK+, AFEW, and DFEW). Extensive experiments and ablation studies show that our proposed DPCNet can vastly improve the performance of video-based FER and achieve state-of-the-art results on the large-scale DFEW dataset. Code is available.
BibTeX
@inproceedings{wang2022dpcnet,
  author = {Yan Wang},
  title = {{DPCNet: Dual Path Multi-Excitation Collaborative Network for Facial Expression Representation Learning in Videos}},
  booktitle = {ACM Multimedia},
  year = {2022},
  doi = {10.1145/3503161.3547865},
}

Loading related content...

Figure