PicoAudio2: Temporal Controllable Text-to-Audio Generation with Natural Language Description

Zihao Zheng, Zeyu Xie, Xuenan Xu, Wen Wu, Chao Zhang, Mengyue WuarXiv 2025 · DOI · arXiv

Abstract

PicoAudio2 is a novel text-to-audio (TTA) framework designed for precise temporal control using natural language descriptions. By integrating a dual-stream architecture that separates global semantics from local timing and a robust data curation pipeline for "temporally-strong" labels, PicoAudio2 achieves state-of-the-art performance in alignment and audio quality. This approach addresses the limitations of existing models that rely on rigid control schemes or lack high-quality timestamped training data.

Introduction

Recent advancements in text-to-audio (TTA) generation have enabled the creation of high-fidelity soundscapes from simple text prompts. However, achieving precise temporal control—such as specifying that a dog should bark from 1 to 3 seconds followed by a man speaking at 5 seconds—remains a significant challenge. Current models face three primary hurdles: a scarcity of "temporally-strong" labels in real-world datasets, the poor generalization of models trained solely on synthetic data, and the use of closed-vocabulary control mechanisms that restrict user flexibility. PicoAudio2 overcomes these issues by enabling fine-grained temporal control through open-ended natural language instructions.

Methodology

The Temporally-Strong Data Pipeline

To address the lack of precise timestamps in high-quality audio datasets, PicoAudio2 introduces a dual-stream data curation pipeline. For simulated data, Large Language Models (LLMs) are utilized to transform categorical labels into diverse natural language descriptions, bridging the gap between synthetic and real-world text. For real-world data, the pipeline employs a Text-to-Audio Grounding (TAG) model to automatically extract precise timestamps from complex captions, effectively creating a large-scale, temporally-strong dataset from existing libraries like AudioCaps.

The PicoAudio2 Data Pipeline illustrating the curation of real and simulated data for training.

Dual-Stream Architecture

The core of PicoAudio2 is a Diffusion Transformer (DiT) backbone that utilizes a dual-stream input strategy to decouple sound content from its timing:

  • Global Semantics: A frozen Flan-T5 encoder processes the overall caption (Temporal-Coarse Caption or TCC) to establish the general acoustic context.
  • Timestamp Roadmap: A dedicated Timestamp Encoder processes detailed event descriptions and their associated timings (Temporal-Detailed Caption or TDC). This information is mapped into a matrix that guides the DiT blocks on which acoustic features to activate at specific time intervals.
The PicoAudio2 architecture featuring the dual-stream input, Timestamp Encoder, and Diffusion Transformer backbone.

Experimental Setup and Results

PicoAudio2 was trained on a comprehensive dataset comprising 64,000 simulated triplets, 49,000 high-quality real-world pairs, and 106,000 weakly labeled pairs. Evaluation was conducted on a specialized subset called AudioCaps-DisJoint (DJ), designed to measure timing precision for non-overlapping events.

The model significantly outperformed baselines such as AudioLDM2 and AudioComposer across multiple metrics. Notably, PicoAudio2 achieved a Segment-F1 score of 0.857, compared to 0.690 for AudioComposer. In multi-event scenarios, its performance (Seg-F1-ME of 0.771) nearly doubled that of several competitors. Human evaluations further confirmed these findings, with PicoAudio2 receiving top marks for both audio quality (3.29 MOS) and temporal controllability (4.15 MOS).

Discussion and Conclusion

A key insight from this work is the importance of decoupling global semantics from local timing. This separation allows the model to handle complex, wordy prompts without losing track of the requested temporal structure. Furthermore, the inclusion of real-world data in the training process was found to be critical for maintaining the natural decay and onset of sounds, which purely synthetic models often fail to capture.

While PicoAudio2 sets a new standard for temporal control, it currently faces challenges with overlapping events due to limitations in the grounding models used for data curation. Future research will focus on enhancing the model's ability to generate complex, layered soundscapes where multiple events occur simultaneously.

Summary updated

Abstract

While recent work in controllable text-to-audio (TTA) generation has achieved fine-grained control through timestamp conditioning, its scope remains limited by audio quality and input format. These models often suffer from poor audio quality in real datasets due to sole reliance on synthetic data. Moreover, some models are constrained to a closed vocabulary of sound events, preventing them from controlling audio generation for open-ended, free-text queries. This paper introduces PicoAudio2, a framework that advances temporal-controllable TTA by mitigating these data and architectural limitations. Specifically, we use a grounding model to annotate event timestamps of real audio-text datasets to curate temporally-strong real data, in addition to simulation data from existing works. The model is trained on the combination of real and simulation data. Moreover, we propose an enhanced architecture that integrates the fine-grained information from a timestamp matrix with coarse-grained free-text input. Experiments show that PicoAudio2 exhibits superior performance in terms of temporal controllability and audio quality.
BibTeX
@misc{zheng2025picoaudio,
  author = {Zihao Zheng and Zeyu Xie and Xuenan Xu and Wen Wu and Chao Zhang and Mengyue Wu},
  title = {{PicoAudio2: Temporal Controllable Text-to-Audio Generation with Natural Language Description}},
  year = {2025},
  eprint = {2509.00683},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2509.00683},
}
Figure