TL;DR
This paper introduces performance conditioning for diffusion-based multi-instrument music synthesis, allowing users to specify not just the notes (MIDI), but also the specific "sonic signature" of a recording—including instrument timbre, performer style, and room acoustics. By using FiLM layers to inject performance IDs into the generative process, the model achieves state-of-the-art realism and significantly reduces "instrument drift" in complex orchestral arrangements.
What Problem Does This Paper Address?
Synthesizing high-fidelity audio from symbolic MIDI files is a long-standing challenge in Music Information Retrieval (MIR). While traditional synthesizers often sound "flat" or "robotic," modern generative models (like Diffusion models) struggle with two specific issues:
- Lack of Control: Users can't easily specify that they want a "1975 Segovia guitar sound" versus a generic acoustic guitar.
- Instrument Drift: In multi-instrument pieces, a model might start a phrase with a violin that gradually morphs into a flute-like sound because it lacks a global "memory" of the specific instrument's character.
The authors aim to bridge this gap by conditioning the synthesis on a specific performance environment, ensuring that the generated audio remains stylistically and acoustically consistent throughout a piece.
Methods
The authors build upon Denoising Diffusion Probabilistic Models (DDPMs) and experiment with two primary architectures: a U-Net (optimized for spectrograms) and a T5 Transformer (the current state-of-the-art).
Key technical components include:
- Performance Conditioning via FiLM: They use Feature-wise Linear Modulation (FiLM) layers to influence the model. A unique "Performance ID" (representing a specific ensemble or recording session) is converted into an embedding and used to apply affine transformations to the network's internal features at every block.
- Spectral Domain Synthesis: The model generates mel-spectrograms, which are then converted to high-fidelity audio using a pre-trained Soundstream vocoder.
- Temporal Coherency: To generate long tracks (minutes instead of seconds), they use an overlapping sampling technique. They generate ~5-second segments with short overlaps and linearly interpolate between them during the diffusion steps to ensure smooth transitions.
- Classifier-Free Guidance (CFG): This technique is used during inference to amplify the influence of both the MIDI notes and the performance condition, allowing for a "stronger" adherence to the requested style.
Datasets
The model was trained on a curated dataset of western classical music:
- Training Set: 197 performances (~58 hours) featuring 19 different instruments, including full symphonies and chamber music. Data was sourced from YouTube and Musopen, with MIDI transcriptions aligned to the audio.
- Evaluation Set: 58 unseen MIDI performances (~5 hours).
- Data Augmentation: They used pitch-shifting (±2 semitones) to increase the variety of the training data.
Key Insights and Ideas
- The "Sonic Identity" is Global: By assigning the same Performance ID to an entire box set of recordings (e.g., a specific orchestra in a specific hall), the model learns to decouple the notes from the acoustic environment.
- FiLM is Efficient: Instead of retraining a model for every instrument or style, FiLM allows a single large model to share parameters across all styles while remaining flexible enough to pivot its output based on a simple ID.
- Reducing Instrument Drift: One of the most significant "aha" moments is that performance conditioning helps the model maintain "instrument identity." When the model "knows" it is supposed to sound like the Berlin Philharmonic, it is much less likely to accidentally swap a trumpet for a clarinet mid-phrase.
Key Results
The authors evaluated their model using the Fréchet Audio Distance (FAD), a metric that measures how "real" generated audio sounds compared to a reference set.
- Realism: The T5 model with performance conditioning improved its TRILL-based FAD score from 0.12 to 0.09, indicating higher perceptual quality.
- Style Adherence: In a classification test, the model's ability to "mimic" a target performance jumped from 36% (without conditioning) to 68% (with conditioning).
- Transcription Accuracy: For complex polyphonic music, the "note-with-instrument" accuracy (getting both the note and the instrument right) improved from 36% to 47% in the T5 model.
- Architecture Trade-offs: While the T5 Transformer achieved the best overall results, the U-Net architecture was significantly faster to train, making it a viable "budget" alternative.
Limitations and Future Work
- Data Scarcity: The method relies on high-quality, time-aligned MIDI and audio pairs. While classical music has some such datasets, genres like jazz, pop, or ethnic music are much harder to source.
- Fixed IDs: Currently, the model uses discrete integer IDs. Future work could involve using continuous embeddings (like a "style vector") to allow for blending between different performances.
- Domain Expansion: The authors suggest this framework could be extended beyond music to human speech, potentially allowing for better control over speaker emotion and recording acoustics in Text-to-Speech (TTS) systems.
Summary updated