Abstract
Instrument playing is among the most common scenes in music-related videos, which represent nowadays one of the largest sources of online videos. In order to understand the instrument-playing scenes in the videos, it is important to know what instruments are played, when they are played, and where the playing actions occur in the scene. While audio-based recognition of instruments has been widely studied, the visual aspect of the music instrument playing remains largely unaddressed in the literature. One of the main obstacles is the difficulty in collecting annotated data of the action locations for training-based methods. To address this issue, we propose a weakly-supervised framework to find when and where the instruments are played in the videos. We propose to use two auxiliary models, a sound model and an object model, to provide supervisions for training the instrument-playing action model. The sound model provides temporal supervisions, while the object model provides spatial supervisions. They together can simultaneously provide temporal and spatial supervisions. The resulted model only needs to analyze the visual part of a music video to deduce which, when and where instruments are played. We found that the proposed method significantly improves the localization accuracy. We evaluate the result of the proposed method temporally and spatially on a small dataset (totally 5,400 frames) that we manually annotated.
Introduction
Identifying musical instruments in video is a multi-faceted task. Beyond simple classification, "action detection" involves pinpointing the spatial coordinates and time intervals of the playing action. Traditional supervised methods require labor-intensive frame-by-frame annotations, such as bounding boxes and timestamps. Labeling even a short clip is incredibly labor-intensive, leading to a lack of large-scale datasets for visual instrument-playing actions. This paper addresses the challenge of training models using only clip-level tags (weak supervision) by leveraging cross-modal cues from sound and object recognition.
Methodology
The researchers utilize two fundamental physical constraints of instrument playing:
- Action-in-object: The playing action occurs within the spatial bounds of the instrument.
- Action-making-sound: The action is temporally synchronized with the resulting audio.
The core architecture is a Fully Convolutional Network (FCN) action model that takes dense optical flow (motion information) as input. To train this model without ground-truth action labels, the authors use two auxiliary "teacher" models: a Sound Model for temporal supervision and an Object Model for spatial supervision.
Four levels of supervision were explored, as shown in the figure below:
- Video Tag (VT): Uses only clip-level labels.
- Object as Target (OT): Uses the object model to provide spatial constraints.
- Sound as Target (ST): Uses the sound model to provide temporal constraints.
- Sound x Object as Target (SOT): Combines both for comprehensive spatio-temporal supervision.
Results and Discussion
The SOT model significantly outperformed simpler variants, achieving a temporal AUC of 0.834 and a spatial error of 29.1 pixels. Further refinement through model fusion—multiplying the outputs of the action, sound, and object models during inference—pushed the temporal AUC to 0.917.
Qualitative results demonstrate the model's ability to distinguish between the instrument itself (object) and the specific region of interaction (action). For example, in violin playing, the object model highlights the entire instrument, while the action model focuses on the bowing motion. The model also successfully learned distinct motion signatures, such as the horizontal movements associated with an accordion versus the diagonal bowing of a cello.
Conclusion
This framework demonstrates that pre-trained models in related domains (audio and object recognition) can effectively "teach" a new model to detect complex actions without manual frame-level labels. While challenges remain with subtle motions (e.g., piano keys) or thin objects (e.g., flutes), the approach provides a scalable path for action detection in diverse video categories where sound and motion are intrinsically linked.
Summary updated