Abstract
FlowSynth addresses the "timbral drift" problem in AI-generated musical instruments by introducing Distributional Flow Matching (DFM). By modeling the velocity field as a Gaussian distribution rather than a deterministic mapping, FlowSynth enables an uncertainty-aware test-time search to select consistent-sounding notes across different pitches and velocities. Experimental results on the NSynth dataset demonstrate that FlowSynth significantly outperforms existing state-of-the-art models like TokenSynth in both audio quality and timbre consistency, offering a scalable approach to high-quality virtual instrument creation.
Introduction
In virtual instrument design, "timbre consistency" is critical: a "warm vintage piano" should maintain its character whether played at middle C or two octaves higher. However, current generative models often suffer from timbral drift, where the instrument's identity changes unpredictably across the keyboard. This occurs because standard Flow Matching or Diffusion models are deterministic during inference, providing a single "best guess" without a mechanism to explore alternative, more consistent sounds.
Methods: Distributional Flow Matching
The core innovation of FlowSynth is Distributional Flow Matching (DFM). Unlike standard flow matching, DFM parameterizes the velocity field as a Gaussian distribution.
Probabilistic Velocity Fields and Training
The model, a 24-layer Diffusion Transformer (DiT), predicts both a mean ($\mu$) and a variance ($\sigma$). It is trained using a Negative Log-Likelihood (NLL) loss, forcing it to learn "problem-specific uncertainty." In regions of high ambiguity, the variance increases, signaling the search algorithm where it has the freedom to explore.
Test-Time Search
During inference, the model leverages this learned uncertainty to sample multiple potential trajectories. These candidates are evaluated against a "timbre consistency objective" using CLAP embeddings. The system selects the candidate that best matches the character of previously generated notes.
Results and Evaluation
FlowSynth was evaluated on the NSynth dataset, focusing on the 88-key piano range. The model sets a new state-of-the-art for controllable instrument generation.
Performance Comparison
Even without test-time search, the base FlowSynth model outperforms the previous leader, TokenSynth, in audio quality (FAD) and timbre consistency. With guided search, the model achieves a 25% lower Timbre Consistency Loss than TokenSynth.
Compute-to-Quality Scaling
The researchers found that increasing the search budget (sampling more candidates) leads to logarithmic improvements in performance. An 8x increase in search budget improved timbre consistency by 13% and prompt adherence by 246%.
Conclusion
FlowSynth demonstrates that explicitly modeling predictive uncertainty allows generative models to solve complex consistency problems through optimization. While the search-based approach introduces some latency—taking approximately 3 minutes for a full 88-key library on an A100 GPU—it remains highly viable for professional studio workflows. Future work will focus on reducing real-time latency and scaling the model to larger, more diverse instrument datasets.
Summary updated