TL;DR
This paper introduces SyncViolinist, a novel framework that generates realistic violin performance motion solely from audio input. By explicitly modeling bowing and fingering, it achieves more accurate and natural body movements than previous audio-driven methods, and introduces a new, richly annotated dataset for violin performance.
What Problem Does This Paper Address?
Creating realistic digital performances, especially for musical instruments, is a significant challenge. Traditional methods like motion capture are expensive and labor-intensive, often requiring extensive manual annotation or specialized equipment. Existing automated approaches for generating musical performance motion from audio often struggle to capture the intricate, fine-grained movements of musicians, particularly for instruments like the violin where precise bowing and fingering are crucial for sound production and expression. These methods often fall short due to the complex mapping between audio and motion, frequently requiring additional inputs like musical scores or MIDI data, which limits their practicality. The core problem is generating synchronized, expressive, and physically accurate violin performance motion using only the audio signal.
Methods
SyncViolinist employs a two-stage, end-to-end framework:
- Bowing/Fingering Module: This module uses Convolutional Recurrent Neural Networks (CRNNs) to extract detailed playing information from the audio's Mel-scaled spectrogram. It predicts four key features: bow direction, played string, finger number, and position. These features are crucial for dictating the nuances of violin performance.
- Motion Generation Module: This module takes both the audio features and the predicted bowing/fingering information as input. It utilizes a novel multi-parallel bidirectional LSTM (BiLSTM) architecture, with separate branches for different body parts (left hand, left arm, right hand & arm, others). This separation accounts for the varying temporal granularity of movements in different body segments, leading to more natural and coordinated global and fine-grained motions. The explicit bowing/fingering information helps constrain the vast possibilities of body motion, guiding the generation towards physically plausible and musically relevant movements.
Datasets
The paper introduces a newly constructed dataset of violin performances. This dataset includes:
- Audio recordings of violin performances.
- Comprehensive 3D body motion data captured using a motion capture system with 57 optical markers (including 20 on fingers).
- Accurately annotated bowing and fingering information (bow direction, played string, finger number, position) derived from musicians annotating musical scores before performing.
This dataset comprises 182.5 minutes of performance data across 61 pieces from three professional violinists. It is noted as having the most pieces and tracked joints among similar datasets.
Key Insights and Ideas
- Explicit Bowing/Fingering is Key: The paper strongly advocates for the explicit modeling of bowing and fingering as critical intermediate representations. This insight is central to achieving realistic violin motion, as these actions directly influence the sound and are intricately linked to body movements.
- Body Semantics for Granularity: Separating motion generation by body parts (left hand, right arm, etc.) is crucial. This acknowledges that different body segments move with different temporal characteristics and allows for more nuanced and coordinated motion generation.
- Audio-Only Generation is Feasible and Desirable: By overcoming the limitations of previous methods, SyncViolinist demonstrates that high-quality, expressive motion generation for complex musical performances can be achieved using only audio input, significantly improving production workflows.
- Marker-Based Data Can Be Natural: Contrary to common criticisms, the authors show through subjective evaluation that their marker-based dataset can yield more natural motion than some markerless approaches, likely due to the higher accuracy and reliability of the ground truth.
Key Results
- Quantitative: SyncViolinist outperformed state-of-the-art baseline methods across multiple metrics (L1 loss, DTW distance, Jerk) for overall body motion, right arm, left arm, and left-hand fingers. For example, the proposed method achieved an L1 score of 9.09, significantly lower than the best baseline at 13.50.
- Qualitative: Subjective evaluations with 46 participants, including professional violinists, indicated that the generated motions were significantly more natural and realistic compared to existing methods.
- Ablation Studies: These studies confirmed the importance of each bowing/fingering feature, with "played string" and "bow direction" showing significant contributions. The multi-branch architecture also demonstrated superior performance over a single-branch BiLSTM.
Limitations and Future Work
- Artist-Specific Styles: The current approach is limited in generating artist-specific nuances, as individual interpretations heavily influence bowing and fingering choices.
- Generalization to Other Instruments: The authors plan to extend their multi-stage approach to other string instruments like cellos and guitars, which also require nuanced global and fine-grained motions.
- Capturing Unique Styles: Future work will focus on exploring methods to capture performers' unique styles for greater expressiveness and customization.
Summary updated