Abstract
Text-based audio generation models have limitations as they cannot encompass all the information in audio, leading to restricted controllability when relying solely on text. To address this issue, we propose a novel model that enhances the controllability of existing pre-trained text-to-audio models by incorporating additional conditions including content (timestamp) and style (pitch contour and energy contour) as supplements to the text. This approach achieves fine-grained control over the temporal order, pitch, and energy of generated audio. To preserve the diversity of generation, we employ a trainable control condition encoder that is enhanced by a large language model and a trainable Fusion-Net to encode and fuse the additional conditions while keeping the weights of the pre-trained text-to-audio model frozen. Due to the lack of suitable datasets and evaluation metrics, we consolidate existing datasets into a new dataset comprising the audio and corresponding conditions and use a series of evaluation metrics to evaluate the controllability performance. Experimental results demonstrate that our model successfully achieves fine-grained control to accomplish controllable audio generation.
Introduction
While current text-to-audio (TTA) models like AudioLDM or Tango generate impressive soundscapes, they suffer from a significant "controllability gap." Natural language is often too blunt to specify precise acoustic details, such as exactly when a sound event should occur or the specific pitch and energy trajectories of a sound. Existing models frequently ignore temporal instructions or specific acoustic attributes mentioned in prompts. Furthermore, retraining these massive models to accept new inputs is computationally prohibitive and risks "catastrophic forgetting" of their original generative capabilities.
Methods
Our proposed framework addresses these challenges by augmenting a frozen, pre-trained TTA model (Tango) with a modular control architecture. This design allows for fine-grained control without sacrificing the diversity of the original model.
The architecture centers on three key components:
- Frozen Foundation: The core Variational Auto-Encoder (VAE) and Latent Diffusion Model (LDM) weights are kept frozen to maintain generative quality.
- Control Condition Encoder: This module processes three supplementary inputs: timestamps, pitch contours, and energy contours. Timestamps are linked to semantic embeddings from a frozen FLAN-T5-LARGE model, ensuring the model understands which sound event corresponds to which time frame.
- Fusion-Net: A trainable "plug-in" consisting of ResNet blocks and self-attention layers that injects control embeddings into the U-Net's intermediate layers during the diffusion process.
The AudioCondition Dataset
To train and evaluate the model, we developed AudioCondition, a dataset combining audio, text captions, and fine-grained control labels. We sourced audio and timestamps from AudioSetStrong (456 sound classes) and high-quality text captions from WavCaps. Pitch and energy contours were extracted directly from the raw audio using signal processing tools. The final dataset comprises approximately 90,000 training samples and 1,100 test samples.
Results and Discussion
Experimental results show that our model significantly outperforms standard TTA baselines in controllability. In Sound Event Detection (SED) evaluations, the model achieved an Event-based F1 score (Eb) of 29.07%, compared to just 5.20% for the base Tango model. For energy control, the model achieved a Mean Absolute Error (MAE) of 0.200, the lowest among all tested configurations.
Key insights from our experiments include:
- Efficiency: The multi-condition model requires only 1076M parameters, whereas three separate single-condition models would require over 3177M.
- Synergy: Training on all three conditions simultaneously improved performance compared to single-condition training, suggesting the model learns a holistic representation of acoustic features.
- Guidance Balancing: A classifier-free guidance scale of 5 was found to be optimal for balancing control adherence with natural audio quality.
Conclusion
This research introduces a novel approach to controllable audio generation by integrating a trainable Fusion-Net into frozen pre-trained diffusion models. By successfully incorporating temporal, pitch, and energy controls, the model bridges the gap between vague text prompts and precise acoustic requirements. Future work will focus on expanding the variety of control conditions and improving pitch control for complex non-speech audio.
Summary updated