Abstract
This paper presents a framework to automate the labelling process for gestures in musical performance videos with a 3D Convolutional Neural Network (CNN). We introduce novel methods to overcome class imbalance and co-existent gestures through a Dynamic Batch Balancing approach and spatial-temporal representations. Our study evaluates 18 categories of gestures across multiple videos of guitar performances. The proposed methods improve gesture identification performance by 12% over previous work, achieving an F1 score of 51% for 18 classes and 72% for 7 super-classes.
Introduction
In musicology, analyzing "expressive gestures"—such as a guitarist’s head lift during a crescendo or the specific vibration of a string—is vital for understanding the link between movement and sound. However, manual labeling is a labor-intensive and subjective process. Automating this task with machine learning faces three primary hurdles:
- Class Imbalance: Common gestures like "normal play" occur far more frequently than rare ones like "upbeat head movement."
- Co-occurrence: Performers often execute multiple gestures simultaneously (e.g., frowning while vibrating a note).
- Temporal Ambiguity: Gestures have distinct envelopes (start, peak, fade-out) rather than being single-frame events.
Methodology
The framework utilizes a 3D-ResNet34 architecture to capture both spatial and temporal features. To address the identified challenges, three key innovations were introduced:
- Dynamic Batch Balance (DBB): Unlike standard oversampling, DBB dynamically adjusts the loss function for each training batch. If a batch is dominated by frequent labels, the loss for rare gestures is scaled up, ensuring the model learns the minority classes.
- Temporal Smoothing: Using a Weighted Moving Average (WMA), labels are treated as temporal envelopes rather than binary frame-level indicators. This smoother representation better aligns with human perception of gesture duration.
- Multimodal Representation: The study explored "Early Fusion" by combining video features with audio descriptors like Mel-frequency cepstral coefficients (MFCCs).
Dataset and Analysis
The dataset comprises four annotated videos of a musician performing Austerity Measures II on a 10-string guitar, totaling approximately 55,000 frames at 25fps. Experts identified 18 distinct gesture categories, which were also mapped into 7 broader "super-classes" (e.g., "Head related action," "Facial Expression") to evaluate model robustness at different granularities.
Key Insights
- Gesture Envelopes: Temporal smoothing acts as a regularization technique, preventing the model from being confused by the precise start or end frames of a gesture.
- Audio Integration Challenges: Surprisingly, adding audio features led to underfitting. This suggests that while audio contains relevant cues, finding an optimal joint representation space for music performance is complex.
- Co-occurrence Patterns: Analyzing the relationships between gestures reveals how certain movements frequently appear together or are mutually exclusive, which is captured in the super-class correlations.
Results
The proposed framework significantly outperformed previous benchmarks:
- 18-Class Problem: Achieved an F1 score of 51%, a 12% improvement over the 39% reported in prior studies.
- Super-class Performance: Accuracy rose to 72% F1 score when using the 7 broader categories.
- Multi-video Validation: Peak performance reached 75% F1 score on an ensemble of all four videos.
Limitations and Future Work
The primary challenge remains generalization. Performance drops significantly (F1 scores of 0.21–0.29) when testing on unseen videos with different lighting or backgrounds. Future research will focus on data augmentation to decouple the performer from the background, refining audio embeddings, and expanding the dataset to include diverse instruments.
Summary updated