FlowSynth: Instrument Generation Through Distributional Flow Matching and Test-Time Search

Qihui Yang, Randal J. Leistikow, Yongyi ZangarXiv 2025 · DOI · arXiv

Abstract

FlowSynth addresses the "timbral drift" problem in AI-generated musical instruments by introducing Distributional Flow Matching (DFM). By modeling the velocity field as a Gaussian distribution rather than a deterministic mapping, FlowSynth enables an uncertainty-aware test-time search to select consistent-sounding notes across different pitches and velocities. Experimental results on the NSynth dataset demonstrate that FlowSynth significantly outperforms existing state-of-the-art models like TokenSynth in both audio quality and timbre consistency, offering a scalable approach to high-quality virtual instrument creation.

Introduction

In virtual instrument design, "timbre consistency" is critical: a "warm vintage piano" should maintain its character whether played at middle C or two octaves higher. However, current generative models often suffer from timbral drift, where the instrument's identity changes unpredictably across the keyboard. This occurs because standard Flow Matching or Diffusion models are deterministic during inference, providing a single "best guess" without a mechanism to explore alternative, more consistent sounds.

Methods: Distributional Flow Matching

The core innovation of FlowSynth is Distributional Flow Matching (DFM). Unlike standard flow matching, DFM parameterizes the velocity field as a Gaussian distribution.

Probabilistic Velocity Fields and Training

The model, a 24-layer Diffusion Transformer (DiT), predicts both a mean ($\mu$) and a variance ($\sigma$). It is trained using a Negative Log-Likelihood (NLL) loss, forcing it to learn "problem-specific uncertainty." In regions of high ambiguity, the variance increases, signaling the search algorithm where it has the freedom to explore.

During inference, the model leverages this learned uncertainty to sample multiple potential trajectories. These candidates are evaluated against a "timbre consistency objective" using CLAP embeddings. The system selects the candidate that best matches the character of previously generated notes.

Overview of FlowSynth training (left) and inference (right) pipelines.

Results and Evaluation

FlowSynth was evaluated on the NSynth dataset, focusing on the 88-key piano range. The model sets a new state-of-the-art for controllable instrument generation.

Performance Comparison

Even without test-time search, the base FlowSynth model outperforms the previous leader, TokenSynth, in audio quality (FAD) and timbre consistency. With guided search, the model achieves a 25% lower Timbre Consistency Loss than TokenSynth.

Quantitative comparison of FlowSynth against Ground Truth and TokenSynth across various metrics.

Compute-to-Quality Scaling

The researchers found that increasing the search budget (sampling more candidates) leads to logarithmic improvements in performance. An 8x increase in search budget improved timbre consistency by 13% and prompt adherence by 246%.

Conclusion

FlowSynth demonstrates that explicitly modeling predictive uncertainty allows generative models to solve complex consistency problems through optimization. While the search-based approach introduces some latency—taking approximately 3 minutes for a full 88-key library on an A100 GPU—it remains highly viable for professional studio workflows. Future work will focus on reducing real-time latency and scaling the model to larger, more diverse instrument datasets.

Summary updated

Abstract

Virtual instrument generation requires maintaining consistent timbre across different pitches and velocities, a challenge that existing note-level models struggle to address. We present FlowSynth, which combines distributional flow matching (DFM) with test-time optimization for high-quality instrument synthesis. Unlike standard flow matching that learns deterministic mappings, DFM parameterizes the velocity field as a Gaussian distribution and optimizes via negative log-likelihood, enabling the model to express uncertainty in its predictions. This probabilistic formulation allows principled test-time search: we sample multiple trajectories weighted by model confidence and select outputs that maximize timbre consistency. FlowSynth outperforms the current state-of-the-art TokenSynth baseline in both single-note quality and cross-note consistency. Our approach demonstrates that modeling predictive uncertainty in flow matching, combined with music-specific consistency objectives, provides an effective path to professional-quality virtual instruments suitable for real-time performance.
BibTeX
@misc{yang2025flowsynth,
  author = {Qihui Yang and Randal J. Leistikow and Yongyi Zang},
  title = {{FlowSynth: Instrument Generation Through Distributional Flow Matching and Test-Time Search}},
  year = {2025},
  eprint = {2510.21667},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2510.21667},
}
Figure