Component attention network for multimodal dance improvisation recognition

Jia Fu, Jiarui Tan, Wenjie Yin, Sepideh Pashami, Mårten Björkman

TL;DR

This paper introduces a novel multimodal fusion network, CANet, for recognizing the expressive qualities of dance improvisation. By leveraging attention mechanisms across different modalities (skeletal data, IMU, audio), their approach effectively captures subtle temporal and component-wise nuances, outperforming existing methods and offering insights into what makes dance movements expressive.

What Problem Does This Paper Address?

Dance improvisation, a core element of contemporary dance, presents a significant challenge for automated analysis due to its inherent spontaneity and dynamic nature. While existing methods often rely solely on skeletal data, this paper argues that incorporating other modalities like audio and Inertial Measurement Unit (IMU) data can enrich the understanding of improvised movements. The key challenge is to effectively fuse these diverse data streams to accurately recognize expressive qualities, a task that has received limited attention in computer vision and machine learning research. The authors aim to bridge this gap by developing a multimodal fusion framework that can identify subtle differences in dance improvisation, potentially aiding in dance education and analysis.

Methods

The paper proposes a Component Attention Network (CANet) as their primary method for multimodal fusion. CANet operates on three levels:

  1. Feature Fusion with CANet: This is the core of their approach. CANet uses a shared LSTM for each modality's temporal processing, followed by a temporal attention module. Crucially, a component attention module then fuses these temporally-attended features, learning to weigh the importance of different modalities (components) for the final classification. This allows the model to dynamically focus on the most relevant information from each data stream.
  2. Model Fusion with CANet and Graph Convolutional Network (GCN): To further enhance the representation of skeletal data, they integrate a GCN. The GCN processes the spatial relationships between body joints, creating a more holistic representation of the skeleton. This GCN-enhanced feature representation is then fed into the CANet for fusion, creating GCN-CANet. This approach aims to better capture the spatio-temporal dynamics of human skeletons.
  3. Late Fusion with a Voting Strategy: As a third fusion strategy, they employ a simple voting mechanism where predictions from multiple independently trained models (each focusing on a specific modality or fusion level) are combined to make a final decision.

Datasets

The authors utilize the Unige-Maastricht Dance dataset. This dataset comprises improvisational dance segments labeled with two expressive qualities: "lightness" and "fragility." The data is recorded in four modalities:

  • Skeletal data: Extracted from video using AlphaPose.
  • IMU data: From sensors on the dancer's left and right hands.
  • Respiration audio: Captured via microphone.
  • Electromyography (EMG) signals: (Note: EMG was abandoned in experiments due to data loss in some segments).

The dataset is preprocessed into 3-second segments with an 80% overlap.

Key Insights and Ideas

  • Multimodality is Crucial for Nuance: The experiments highlight that while skeletal data is informative, fusing it with IMU and audio data significantly improves recognition accuracy, especially for subtle expressive qualities.
  • Attention is Key to Understanding: The component attention mechanism in CANet allows the model to dynamically identify which modalities are most important for distinguishing between "lightness" and "fragility" at any given moment.
  • GCN Enhances Skeletal Representation: Integrating GCNs with CANet (GCN-CANet) proves beneficial, suggesting that explicitly modeling the spatial relationships between body joints leads to a more robust understanding of movement dynamics.
  • MFCCs are Tricky for Short Segments: The paper notes that MFCCs (Mel-frequency cepstral coefficients) from audio data sometimes degrade performance, particularly in short segments, as clear breath rhythms might not be captured consistently. This suggests that the temporal scope of audio features needs careful consideration.
  • Visualizing Attention Reveals Expressive Cues: The heatmaps of temporal and component attention scores provide valuable qualitative insights, showing how the model focuses on specific body parts or modalities when recognizing different expressive qualities. For instance, "lightness" might involve more varied attention across limbs, while "fragility" shows more consistent attention across components.

Key Results

  • CANet Outperforms Baselines: The proposed CANet and GCN-CANet models achieve higher classification accuracy than two baseline methods (Fusion-GCN and Gimme Signals) across various fusion configurations (unimodal, bimodal, trimodal).
  • Model Fusion is Preferable: GCN-CANet (model fusion) generally outperforms plain CANet (feature fusion) when using the same training data, indicating the benefit of integrating GCN's spatial understanding.
  • Late Fusion Benefits from MFCC: While MFCCs can be problematic for feature/model fusion, they become necessary and beneficial for late fusion (decision voting), suggesting different fusion strategies have varying sensitivities to modality characteristics.
  • Specific Accuracies: For instance, the GCN-CANet model with Joints + IMU data achieved an accuracy of 83.12% and an F1 score of 0.83. Late fusion with all three modalities (J, I, M) achieved 78.90% accuracy.

Limitations and Future Work

  • MFCC Deficiency: The paper acknowledges that MFCCs can be less effective in short segments due to inconsistent breath rhythm capture. Future work could explore alternative audio features or methods to better capture temporal audio dynamics.
  • Data Limitations: The study focuses on two specific expressive qualities ("lightness" and "fragility"). Expanding to a wider range of expressive qualities or dance styles would be a natural next step.
  • Computational Cost: While not explicitly detailed, multimodal fusion with attention and GCNs can be computationally intensive. Optimizing these models for real-time applications could be a future direction.
  • Generalizability: Further validation on larger and more diverse dance improvisation datasets would be beneficial to confirm the generalizability of their proposed methods.

Summary updated

Abstract

Dance improvisation is an active research topic in the arts. Motion analysis of improvised dance can be challenging due to its unique dynamics. Data-driven dance motion analysis, including recognition and generation, is often limited to skeletal data. However, data of other modalities, such as audio, can be recorded and benefit downstream tasks. This paper explores the application and performance of multimodal fusion methods for human motion recognition in the context of dance improvisation. We propose an attention-based model, component attention network (CANet), for multimodal fusion on three levels: 1) feature fusion with CANet, 2) model fusion with CANet and graph convolutional network (GCN), and 3) late fusion with a voting strategy. We conduct thorough experiments to analyze the impact of each modality in different fusion methods and distinguish critical temporal or component features. We show that our proposed model outperforms the two baseline methods, demonstrating its potential for analyzing improvisation in dance.
BibTeX
@inproceedings{fu2023component,
  author = {Jia Fu and Jiarui Tan and Wenjie Yin and Sepideh Pashami and Mårten Björkman},
  title = {{Component attention network for multimodal dance improvisation recognition}},
  booktitle = {ICMI},
  year = {2023},
  doi = {10.1145/3577190.3614114},
}
Figure