TokenSynth: A Token-based Neural Synthesizer for Instrument Cloning and Text-to-Instrument

Kyungsu Kim, Junghyun Koo, Sungho Lee, Haesun Joung, Kyogu Lee

TL;DR

TokenSynth is a polyphonic neural synthesizer that uses a transformer-based language model to generate audio from MIDI and timbre descriptions. By leveraging cross-modal embeddings (CLAP) and neural audio codecs (DAC), it enables zero-shot instrument cloning from short audio clips and text-to-instrument synthesis without any model fine-tuning.

What Problem Does This Paper Address?

While generative AI has made massive strides in creating full musical compositions, these models often act as "black boxes" with limited user control over individual tracks. In professional music production, instruments are the fundamental building blocks.

Existing solutions for instrument-level control have significant drawbacks:

  1. Monophonic Limits: Many "cloning" methods (like DDSP) are restricted to single-note melodies.
  2. Fine-tuning Requirements: Most models require expensive retraining or fine-tuning to learn a specific new instrument.
  3. Lack of Expressivity: Text-to-instrument models often generate isolated samples rather than expressive, polyphonic performances that follow complex MIDI data.

TokenSynth aims to bridge this gap by creating a flexible, polyphonic synthesizer that can adopt any timbre—whether described by a text prompt or a 5-second audio snippet—and apply it to any MIDI sequence.

Methods

The authors treat music synthesis as a language modeling problem. The architecture is a decoder-only transformer that predicts a sequence of audio tokens.

  • Audio Tokenization: The model uses the Descript Audio Codec (DAC) to compress raw audio into discrete tokens. This allows the transformer to "write" audio one step at a time.
  • MIDI Tokenization: MIDI data (pitch, velocity, onset, and offset) is converted into tokens following the MT3 framework, providing the structural "score" for the model to follow.
  • Timbre Conditioning: This is the core innovation. The model is conditioned on CLAP (Contrastive Language-Audio Pretraining) embeddings. Because CLAP maps both audio and text into the same mathematical space, the model can be "told" what instrument to use via a reference audio clip (cloning) or a text description (synthesis).
  • First-Note Guidance: To improve stability, the authors introduced a novel inference technique. They apply Classifier-Free Guidance (CFG) only at the onset of the first note. This "locks in" the instrument's identity at the start without introducing the artifacts or noise that often occur when applying CFG to silent parts of a sequence.

Datasets

The researchers created a massive synthetic dataset to train the model:

  • NSynth + Lakh MIDI: They combined the NSynth instrument library with Lakh MIDI sequences to generate 9.53 million MIDI-audio pairs.
  • Data Augmentation: To make the model robust, they applied a digital effect chain (EQ, distortion, and reverb) to the audio, doubling the dataset size. This helps the model generalize to "wet" studio recordings rather than just "dry" samples.

Key Insights and Ideas

  • Zero-Shot Versatility: Because the model uses a shared audio-text embedding space (CLAP), it doesn't need to be retrained to learn a new instrument. It "understands" timbre as a general concept.
  • Timbre Interpolation: One of the most exciting "aha moments" is the ability to blend timbres. By interpolating between two CLAP embeddings (e.g., 70% "piano" and 30% "flute" text prompt), users can create entirely new, hybrid instruments.
  • The Power of Onsets: The success of "First-Note Guidance" suggests that the model establishes the "timbral state" at the very beginning of a performance. Once the instrument is established at the first note, the transformer's self-attention mechanism is strong enough to maintain that consistency throughout the clip.

Key Results

The authors evaluated the model on synthesis accuracy (F-score), spectral loss (MSS), and timbral similarity (CLAP score).

  • Instrument Cloning: TokenSynth achieved high timbral similarity, even outperforming ground-truth recordings in some CLAP score metrics. It successfully cloned timbres from unseen instruments using only 5-second references.
  • Text-to-Instrument: While the "modality gap" (the inherent difference between how models see text vs. audio) made text-to-instrument slightly less accurate than audio cloning, the model still followed text prompts effectively.
  • Effect of Augmentation: Training on augmented ("wet") data significantly improved the model's ability to follow MIDI accurately (higher F-score), though it slightly increased spectral loss compared to models trained only on "dry" audio.

Limitations and Future Work

  • Not Real-Time: Currently, the model requires the entire MIDI sequence to be provided upfront, making it unsuitable for live performance.
  • Stochastic Errors: Like all autoregressive models, it can occasionally "hallucinate" or miss a MIDI note due to the random nature of sampling.
  • Velocity Resolution: The model is currently limited to only four velocity levels (inherited from the NSynth dataset), which limits the dynamic range of the performances.
  • Future Directions: The authors suggest moving toward real-time synthesis and improving the precision of velocity and expressive control.

Summary updated

Abstract

Recent advancements in neural audio codecs have enabled the use of tokenized audio representations in various audio generation tasks, such as text-to-speech, text-to-audio, and text-to-music generation. Leveraging this approach, we propose TokenSynth, a novel neural synthesizer that utilizes a decoder-only transformer to generate desired audio tokens from MIDI tokens and CLAP (Contrastive Language-Audio Pretraining) embedding, which has timbre-related information. Our model is capable of performing instrument cloning, text-to-instrument synthesis, and text-guided timbre manipulation without any fine-tuning. This flexibility enables diverse sound design and intuitive timbre control. We evaluated the quality of the synthesized audio, the timbral similarity between synthesized and target audio/text, and synthesis accuracy (i.e., how accurately it follows the input MIDI) using objective measures.TokenSynth demonstrates the potential of leveraging advanced neural audio codecs and transformers to create powerful and versatile neural synthesizers. The source code, model weights, and audio demos are available at: https://github.com/KyungsuKim42/tokensynth
BibTeX
@inproceedings{kim2025tokensynth,
  author = {Kyungsu Kim and Junghyun Koo and Sungho Lee and Haesun Joung and Kyogu Lee},
  title = {{TokenSynth: A Token-based Neural Synthesizer for Instrument Cloning and Text-to-Instrument}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2025},
  url = {https://openreview.net/forum?id=nw4HpCyULx},
}

Loading related content...

Figure