TA-CNN: A Unified Network for Human Behavior Analysis in Multi-Person Conversations

Fuyan Ma, Ziyu Ma, Bin Sun, Shutao LiACM Multimedia 2022 · OpenReview

Abstract

Human behavior analysis in multi-person conversations has been one of the most important research issues for natural human-robot interaction. However, previous datasets and studies mainly focus on single-person behavior analysis, therefore, can hardly be generalized in real-world application scenarios. Fortunately, the MultiMediate'22 Challenge provides various video clips of multi-party conversations. In this paper, we present a unified network named TA-CNN for both sub-challenges. Our TA-CNN can not only model the spatio-temporal dependencies for eye contact detection, but also capture the group-level discriminative features for multi-label next speaker prediction. We empirically evaluate the performance of our method on the officially provided datasets. Our method achieves the state-of-the-art result of 0.7261 for eye contact detection in terms of accuracy and the UAR of 0.5965 for next speaker prediction on the corresponding test sets.
BibTeX
@inproceedings{ma2022ta,
  author = {Fuyan Ma and Ziyu Ma and Bin Sun and Shutao Li},
  title = {{TA-CNN: A Unified Network for Human Behavior Analysis in Multi-Person Conversations}},
  booktitle = {ACM Multimedia},
  year = {2022},
  url = {https://openreview.net/forum?id=3gjEwmOu0x},
}

Loading related content...

Figure