Synthesizing a convincing acoustic guitar is a unique challenge in the field of neural audio synthesis. Unlike the piano, where a MIDI note translates relatively directly to a sound, the guitar’s timbre is shaped by the complex interplay of six strings, varying fret positions (tablature), and subtle performance nuances such as pick noise and micro-timing during strums. Researchers from KAIST have addressed these challenges by introducing a diffusion-based model that utilizes a novel "guitarroll" representation and a diffusion outpainting technique to ensure long-term consistency and realistic timbre.
The Challenge of Guitar Synthesis
Existing neural audio synthesis models often struggle with two primary issues:
- Input Representation: Standard MIDI or "pianoroll" representations are extremely sparse and fail to capture the physical constraints of the instrument—specifically, which string is producing which note.
- Long-term Coherence: Generating long passages of audio often leads to "drifting" or audible seams between segments. Previous diffusion models attempted to fix this by concatenating features from previous segments, but this is often computationally inefficient and ineffective at maintaining a natural flow.
Methods: Guitarroll and Diffusion Outpainting
The authors introduce a framework that combines a novel input format with a streamlined diffusion architecture designed for instrument-specific synthesis.
Guitarroll Representation
Instead of a massive, sparse grid of all possible MIDI notes, the researchers proposed the guitarroll. This is a matrix of size $6 \times L$ (where 6 represents the strings and $L$ is time). Each cell contains the pitch played on that specific string. This provides a much denser, instrument-specific "inductive bias," allowing the model to learn the relationship between notes and string timbre much more effectively.
Diffusion Outpainting
To generate long-form audio, the model employs an outpainting algorithm based on the RePaint method. Rather than just using a "context" feature, the model treats the generation of a new 2.56-second audio block as a completion task. By using the previous segment as a fixed boundary condition for the future, the model ensures that the new block blends perfectly with the previous one, leading to much smoother transitions in long compositions.
Architecture
The core of the system is a T5-based transformer that acts as a denoiser for mel spectrograms. These spectrograms are then converted into high-fidelity audio using a pre-trained Soundstream vocoder.
Data Strategy: Synthetic Pre-training
A significant hurdle in this research was the data bottleneck; the gold-standard GuitarSet contains only 3 hours of high-quality, MIDI-aligned recordings. To solve this, the researchers created a new benchmark:
- Lakh-Ilya Dataset: A massive, 100-hour dataset created by taking MIDI from the Lakh MIDI Dataset and synthesizing it using a high-quality, sample-based virtual instrument (Ilya Efimov Acoustic Guitar).
- Fine-tuning: The model was pre-trained on the Lakh-Ilya set and then fine-tuned on GuitarSet to capture real-world human performance nuances.
Key Results
The proposed model, OutpaintFT-guitarroll, was compared against a downsized version of the "Hawthorne" model (a previous state-of-the-art work from Google Magenta).
- Objective Metrics: The model achieved a lower Fréchet Audio Distance (FAD)—a measure of how close the generated audio distribution is to real audio—scoring 4.27 compared to the baseline's 4.72.
- Transcription Accuracy: Using an MT3 transcription model to verify the output, the authors found their model had a higher F1 score (0.531), meaning the synthesized audio more accurately reflected the input MIDI.
- Subjective Quality: In listening tests, the model consistently scored higher in Timbre and Overall Quality compared to the baseline, particularly for solo guitar passages.
Limitations and Future Work
While the model excels at timbre and solo melodies, the authors noted that expressiveness in complex chord accompaniments ("comping") still lags slightly behind the best sample-based synthesizers. This is likely because the training data lacks detailed velocity (loudness) and envelope information. Future work will focus on incorporating more expressive control parameters—such as vibrato, slides, and percussive hits—and exploring ways to capture the "groove" of human performance more accurately within the diffusion process.
Summary updated