Abstract
Text-based audio generation models have limitations as they cannot encompass all the information in audio, leading to restricted controllability when relying solely on text. To address this issue, we propose a novel model that enhances the controllability of existing pre-trained text-to-audio models by incorporating additional conditions including content (timestamp) and style (pitch contour and energy contour) as supplements to the text. This approach achieves fine-grained control over the temporal order, pitch, and energy of generated audio. To preserve the diversity of generation, we employ a trainable control condition encoder that is enhanced by a large language model and a trainable Fusion-Net to encode and fuse the additional conditions while keeping the weights of the pre-trained text-to-audio model frozen. Experimental results demonstrate that our model successfully achieves fine-grained control to accomplish controllable audio generation.
Introduction
While current Text-to-Audio (TTA) models like AudioLDM or Tango can generate impressive soundscapes from a prompt like "a dog barking and a car passing by," they offer very little "director-level" control. You can’t easily tell the model exactly when the dog should bark or how the pitch of the car engine should shift. This lack of precision is a major hurdle for applications like film scoring, VR, and game development, where audio must be tightly synchronized with visual events. Language alone is too ambiguous to convey these fine-grained details, necessitating a system that accepts multi-modal "control conditions."
Methodology
The proposed architecture augments existing TTA models rather than replacing them. The approach rests on three modular pillars:
- Frozen Foundation: The system utilizes a pre-trained TTA model (Tango) with frozen weights. This ensures the model retains its high-quality audio generation capabilities learned from massive datasets while preventing "catastrophic forgetting."
- Control Condition Encoder: This module processes three distinct inputs: Timestamps (using FLAN-T5 to create semantic embeddings), Pitch Contour (decomposed via wavelet transforms), and Energy Contour (calculated via STFT).
- Fusion-Net: This trainable network is inserted between the layers of the frozen U-Net. It uses a combination of ResNet blocks and self-attention to fuse control embeddings with the latent audio representation during the denoising process.
Datasets and Evaluation
Because no existing dataset contained audio, high-quality captions, and precise temporal/stylistic metadata, the researchers created AudioCondition. This dataset combines source audio from AudioSetStrong (providing frame-level timestamps) with captions from WavCaps (using ChatGPT for descriptive text). The final dataset includes roughly 90,000 training samples.
Key Results
The model, referred to as Ours, significantly outperformed baselines in both objective and subjective tests across all control dimensions:
- Temporal Control: In event-based measures (Eb), the model achieved 29.07%, nearly triple the performance of AudioLDM (9.75%) and Tango (5.20%).
- Pitch Accuracy: The Dynamic Time Warping (DTW) distance—a measure of how closely the generated pitch matches the target—was 10.79, compared to over 36.0 for standard TTA models.
- Energy Control: The Mean Absolute Error (MAE) for energy was reduced to 0.200, indicating very tight adherence to the requested volume contours.
One of the most interesting findings was that training the model on multiple conditions simultaneously (pitch, energy, and timing) actually improved performance compared to training on a single condition. The model learns a more robust representation of how these features interact. Furthermore, despite supporting more features, the model uses significantly fewer trainable parameters (1076M) than a collection of single-condition models (3177M).
Conclusion
This research demonstrates that fine-grained audio control can be achieved by augmenting frozen pre-trained models with specialized encoders and a fusion network. By moving beyond simple text prompts to include timestamps, pitch, and energy contours, the model provides the precision required for professional audio production. Future work aims to expand the range of control conditions further, potentially including spatial audio positioning and more complex acoustic environments.
Summary updated