Abstract
Recent advancements in neural audio codecs have enabled the use of tokenized audio representations in various audio generation tasks. We propose TokenSynth, a novel neural synthesizer that utilizes a decoder-only transformer to generate audio tokens from MIDI tokens and CLAP (Contrastive Language-Audio Pretraining) embeddings. Our model is capable of performing zero-shot instrument cloning, text-to-instrument synthesis, and text-guided timbre manipulation without any fine-tuning. Evaluation shows high synthesis accuracy and timbral similarity, demonstrating the potential of transformer-based architectures for versatile neural synthesis.
Introduction
In professional music production, the ability to control individual instrument tracks with precision is paramount. However, many current generative AI models for music act as "black boxes," offering limited control over specific timbres or polyphonic performances. Existing solutions often suffer from monophonic limitations, require expensive fine-tuning for new instruments, or lack the expressivity needed for complex MIDI scores.
TokenSynth addresses these challenges by treating music synthesis as a conditional language modeling task. It enables polyphonic synthesis that can adopt any timbre—defined by either a short audio reference or a text prompt—without any model fine-tuning.
Methodology
TokenSynth employs a decoder-only transformer architecture designed to predict sequences of discrete audio tokens based on musical and timbral conditioning.
Tokenization and Conditioning
The system relies on three primary components:
- Audio Representation: The Descript Audio Codec (DAC) compresses raw audio into discrete tokens, allowing the transformer to generate audio autoregressively.
- MIDI Input: MIDI data (pitch, velocity, onset, offset) is tokenized using the MT3 framework, providing the structural backbone for the synthesis.
- Timbre Conditioning: The model is conditioned on CLAP embeddings. Since CLAP maps audio and text into a shared latent space, the model can interpret timbre from either modality.
First-Note Guidance
To ensure timbral consistency, the authors utilize Classifier-Free Guidance (CFG) specifically at the onset of the first note. This "locks in" the instrument's identity at the start of the sequence, leveraging the transformer's self-attention to maintain that identity throughout the performance without introducing artifacts during silent intervals.
Data Augmentation and Training
The model was trained on a massive synthetic dataset combining NSynth instrument samples with Lakh MIDI sequences, totaling 9.53 million pairs. To improve robustness, the researchers applied a digital effect chain (EQ, distortion, reverb) to create an augmented version of the dataset ("TokenSynth-Aug"). This allows the model to generalize better to "wet" studio recordings compared to models trained only on "dry" samples.
Experimental Results
The model was evaluated on synthesis accuracy (F-score), spectral loss (MSS), and timbral similarity (CLAP score).
Instrument Cloning and Synthesis Accuracy
TokenSynth demonstrates remarkable zero-shot cloning capabilities. As shown in the results, the model often achieves CLAP scores comparable to or even exceeding ground-truth recordings, particularly in "dry" conditions.
A key finding is the impact of data augmentation. While the base TokenSynth model performs well on spectral similarity, the augmented model (TokenSynth-Aug) shows a dramatic improvement in MIDI following accuracy (F-score), jumping from approximately 0.64 to over 0.83. This suggests that exposure to varied acoustic environments helps the model better decouple timbre from musical structure.
Text-to-Instrument and Interpolation
While a "modality gap" exists between text and audio embeddings, TokenSynth successfully synthesizes instruments from text descriptions. Furthermore, the model supports timbre interpolation, allowing users to blend embeddings (e.g., mixing "piano" and "flute" prompts) to create hybrid instruments.
Conclusion
TokenSynth represents a significant step toward intuitive and flexible neural synthesis. By combining the power of neural audio codecs with cross-modal embeddings, it provides a versatile tool for instrument cloning and text-guided sound design. Future work will focus on achieving real-time performance and increasing the resolution of expressive controls like velocity.
Summary updated