Audio Generation with Multiple Conditional Diffusion Model

Zhifang Guo, Jianguo Mao, Rui Tao, Long Yan, Kazushige Ouchi, Hong Liu, Xiangdong Wang

Abstract

Text-based audio generation models have limitations as they cannot encompass all the information in audio, leading to restricted controllability when relying solely on text. To address this issue, we propose a novel model that enhances the controllability of existing pre-trained text-to-audio models by incorporating additional conditions including content (timestamp) and style (pitch contour and energy contour) as supplements to the text. This approach achieves fine-grained control over the temporal order, pitch, and energy of generated audio. To preserve the diversity of generation, we employ a trainable control condition encoder that is enhanced by a large language model and a trainable Fusion-Net to encode and fuse the additional conditions while keeping the weights of the pre-trained text-to-audio model frozen. Experimental results demonstrate that our model successfully achieves fine-grained control to accomplish controllable audio generation.

Introduction

While current Text-to-Audio (TTA) models like AudioLDM or Tango can generate impressive soundscapes from a prompt like "a dog barking and a car passing by," they offer very little "director-level" control. You can’t easily tell the model exactly when the dog should bark or how the pitch of the car engine should shift. This lack of precision is a major hurdle for applications like film scoring, VR, and game development, where audio must be tightly synchronized with visual events. Language alone is too ambiguous to convey these fine-grained details, necessitating a system that accepts multi-modal "control conditions."

Methodology

The proposed architecture augments existing TTA models rather than replacing them. The approach rests on three modular pillars:

  1. Frozen Foundation: The system utilizes a pre-trained TTA model (Tango) with frozen weights. This ensures the model retains its high-quality audio generation capabilities learned from massive datasets while preventing "catastrophic forgetting."
  2. Control Condition Encoder: This module processes three distinct inputs: Timestamps (using FLAN-T5 to create semantic embeddings), Pitch Contour (decomposed via wavelet transforms), and Energy Contour (calculated via STFT).
  3. Fusion-Net: This trainable network is inserted between the layers of the frozen U-Net. It uses a combination of ResNet blocks and self-attention to fuse control embeddings with the latent audio representation during the denoising process.
Overview of the proposed model architecture, including the Control Condition Encoder and Fusion-Net.

Datasets and Evaluation

Because no existing dataset contained audio, high-quality captions, and precise temporal/stylistic metadata, the researchers created AudioCondition. This dataset combines source audio from AudioSetStrong (providing frame-level timestamps) with captions from WavCaps (using ChatGPT for descriptive text). The final dataset includes roughly 90,000 training samples.

Key Results

The model, referred to as Ours, significantly outperformed baselines in both objective and subjective tests across all control dimensions:

  • Temporal Control: In event-based measures (Eb), the model achieved 29.07%, nearly triple the performance of AudioLDM (9.75%) and Tango (5.20%).
  • Pitch Accuracy: The Dynamic Time Warping (DTW) distance—a measure of how closely the generated pitch matches the target—was 10.79, compared to over 36.0 for standard TTA models.
  • Energy Control: The Mean Absolute Error (MAE) for energy was reduced to 0.200, indicating very tight adherence to the requested volume contours.
Quantitative comparison of the proposed model against state-of-the-art baselines across temporal, pitch, and energy metrics.

One of the most interesting findings was that training the model on multiple conditions simultaneously (pitch, energy, and timing) actually improved performance compared to training on a single condition. The model learns a more robust representation of how these features interact. Furthermore, despite supporting more features, the model uses significantly fewer trainable parameters (1076M) than a collection of single-condition models (3177M).

Conclusion

This research demonstrates that fine-grained audio control can be achieved by augmenting frozen pre-trained models with specialized encoders and a fusion network. By moving beyond simple text prompts to include timestamps, pitch, and energy contours, the model provides the precision required for professional audio production. Future work aims to expand the range of control conditions further, potentially including spatial audio positioning and more complex acoustic environments.

Summary updated

Abstract

Text-based audio generation models have limitations as they cannot encompass all the information in audio, leading to restricted controllability when relying solely on text. To address this issue, we propose a novel model that enhances the controllability of existing pre-trained text-to-audio models by incorporating additional conditions including content (timestamp) and style (pitch contour and energy contour) as supplements to the text. This approach achieves fine-grained control over the temporal order, pitch, and energy of generated audio. To preserve the diversity of generation, we employ a trainable control condition encoder that is enhanced by a large language model and a trainable Fusion-Net to encode and fuse the additional conditions while keeping the weights of the pre-trained text-to-audio model frozen. Due to the lack of suitable datasets and evaluation metrics, we consolidate existing datasets into a new dataset comprising the audio and corresponding conditions and use a series of evaluation metrics to evaluate the controllability performance. Experimental results demonstrate that our model successfully achieves fine-grained control to accomplish controllable audio generation.
BibTeX
@inproceedings{guo2024audio,
  author = {Zhifang Guo and Jianguo Mao and Rui Tao and Long Yan and Kazushige Ouchi and Hong Liu and Xiangdong Wang},
  title = {{Audio Generation with Multiple Conditional Diffusion Model}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  url = {https://openreview.net/forum?id=6Fj8phklbg},
}
Figure