Deep Neural Network approaches for Analysing Videos of Music Performances

Foteini Simistira Liwicki, Richa Upadhyay, Prakash Chandra Chhipa, Killian Murphy, Federico Visi, Stefan Östersjö, Marcus LiwickiarXiv 2022 · OpenReview

Abstract

This paper presents a framework to automate the labelling process for gestures in musical performance videos with a 3D Convolutional Neural Network (CNN). We introduce novel methods to overcome class imbalance and co-existent gestures through a Dynamic Batch Balancing approach and spatial-temporal representations. Our study evaluates 18 categories of gestures across multiple videos of guitar performances. The proposed methods improve gesture identification performance by 12% over previous work, achieving an F1 score of 51% for 18 classes and 72% for 7 super-classes.

Introduction

In musicology, analyzing "expressive gestures"—such as a guitarist’s head lift during a crescendo or the specific vibration of a string—is vital for understanding the link between movement and sound. However, manual labeling is a labor-intensive and subjective process. Automating this task with machine learning faces three primary hurdles:

  1. Class Imbalance: Common gestures like "normal play" occur far more frequently than rare ones like "upbeat head movement."
  2. Co-occurrence: Performers often execute multiple gestures simultaneously (e.g., frowning while vibrating a note).
  3. Temporal Ambiguity: Gestures have distinct envelopes (start, peak, fade-out) rather than being single-frame events.

Methodology

The framework utilizes a 3D-ResNet34 architecture to capture both spatial and temporal features. To address the identified challenges, three key innovations were introduced:

  • Dynamic Batch Balance (DBB): Unlike standard oversampling, DBB dynamically adjusts the loss function for each training batch. If a batch is dominated by frequent labels, the loss for rare gestures is scaled up, ensuring the model learns the minority classes.
  • Temporal Smoothing: Using a Weighted Moving Average (WMA), labels are treated as temporal envelopes rather than binary frame-level indicators. This smoother representation better aligns with human perception of gesture duration.
  • Multimodal Representation: The study explored "Early Fusion" by combining video features with audio descriptors like Mel-frequency cepstral coefficients (MFCCs).

Dataset and Analysis

The dataset comprises four annotated videos of a musician performing Austerity Measures II on a 10-string guitar, totaling approximately 55,000 frames at 25fps. Experts identified 18 distinct gesture categories, which were also mapped into 7 broader "super-classes" (e.g., "Head related action," "Facial Expression") to evaluate model robustness at different granularities.

Table 1: Frame distribution across classes, highlighting the extreme class imbalance where 'Normal play' and 'Frowning' dominate.

Key Insights

  • Gesture Envelopes: Temporal smoothing acts as a regularization technique, preventing the model from being confused by the precise start or end frames of a gesture.
  • Audio Integration Challenges: Surprisingly, adding audio features led to underfitting. This suggests that while audio contains relevant cues, finding an optimal joint representation space for music performance is complex.
  • Co-occurrence Patterns: Analyzing the relationships between gestures reveals how certain movements frequently appear together or are mutually exclusive, which is captured in the super-class correlations.
Figure 1: Correlation matrix for the 7 super-classes, illustrating how different gesture types relate to one another.

Results

The proposed framework significantly outperformed previous benchmarks:

  • 18-Class Problem: Achieved an F1 score of 51%, a 12% improvement over the 39% reported in prior studies.
  • Super-class Performance: Accuracy rose to 72% F1 score when using the 7 broader categories.
  • Multi-video Validation: Peak performance reached 75% F1 score on an ensemble of all four videos.

Limitations and Future Work

The primary challenge remains generalization. Performance drops significantly (F1 scores of 0.21–0.29) when testing on unseen videos with different lighting or backgrounds. Future research will focus on data augmentation to decouple the performer from the background, refining audio embeddings, and expanding the dataset to include diverse instruments.

Summary updated

Abstract

This paper presents a framework to automate the labelling process for gestures in musical performance videos with a 3D Convolutional Neural Network (CNN). While this idea was proposed in a previous study, this paper introduces several novelties: (i) Presents a novel method to overcome the class imbalance challenge and make learning possible for co-existent gestures by batch balancing approach and spatial-temporal representations of gestures. (ii) Performs a detailed study on 7 and 18 categories of gestures generated during the performance (guitar play) of musical pieces that have been video-recorded. (iii) Investigates the possibility to use audio features. (iv) Extends the analysis to multiple videos. The novel methods significantly improve the performance of gesture identification by 12 %, when compared to the previous work (51 % in this study over 39 % in previous work). We successfully validate the proposed methods on 7 super classes (72 %), an ensemble of the 18 gestures/classes, and additional videos (75 %).
BibTeX
@misc{liwicki2022deep,
  author = {Foteini Simistira Liwicki and Richa Upadhyay and Prakash Chandra Chhipa and Killian Murphy and Federico Visi and Stefan Östersjö and Marcus Liwicki},
  title = {{Deep Neural Network approaches for Analysing Videos of Music Performances}},
  year = {2022},
  eprint = {2205.11232},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2205.11232},
}
Figure