Abstract
PicoAudio2 is a novel text-to-audio (TTA) framework designed for precise temporal control using natural language descriptions. By integrating a dual-stream architecture that separates global semantics from local timing and a robust data curation pipeline for "temporally-strong" labels, PicoAudio2 achieves state-of-the-art performance in alignment and audio quality. This approach addresses the limitations of existing models that rely on rigid control schemes or lack high-quality timestamped training data.
Introduction
Recent advancements in text-to-audio (TTA) generation have enabled the creation of high-fidelity soundscapes from simple text prompts. However, achieving precise temporal control—such as specifying that a dog should bark from 1 to 3 seconds followed by a man speaking at 5 seconds—remains a significant challenge. Current models face three primary hurdles: a scarcity of "temporally-strong" labels in real-world datasets, the poor generalization of models trained solely on synthetic data, and the use of closed-vocabulary control mechanisms that restrict user flexibility. PicoAudio2 overcomes these issues by enabling fine-grained temporal control through open-ended natural language instructions.
Methodology
The Temporally-Strong Data Pipeline
To address the lack of precise timestamps in high-quality audio datasets, PicoAudio2 introduces a dual-stream data curation pipeline. For simulated data, Large Language Models (LLMs) are utilized to transform categorical labels into diverse natural language descriptions, bridging the gap between synthetic and real-world text. For real-world data, the pipeline employs a Text-to-Audio Grounding (TAG) model to automatically extract precise timestamps from complex captions, effectively creating a large-scale, temporally-strong dataset from existing libraries like AudioCaps.
Dual-Stream Architecture
The core of PicoAudio2 is a Diffusion Transformer (DiT) backbone that utilizes a dual-stream input strategy to decouple sound content from its timing:
- Global Semantics: A frozen Flan-T5 encoder processes the overall caption (Temporal-Coarse Caption or TCC) to establish the general acoustic context.
- Timestamp Roadmap: A dedicated Timestamp Encoder processes detailed event descriptions and their associated timings (Temporal-Detailed Caption or TDC). This information is mapped into a matrix that guides the DiT blocks on which acoustic features to activate at specific time intervals.
Experimental Setup and Results
PicoAudio2 was trained on a comprehensive dataset comprising 64,000 simulated triplets, 49,000 high-quality real-world pairs, and 106,000 weakly labeled pairs. Evaluation was conducted on a specialized subset called AudioCaps-DisJoint (DJ), designed to measure timing precision for non-overlapping events.
The model significantly outperformed baselines such as AudioLDM2 and AudioComposer across multiple metrics. Notably, PicoAudio2 achieved a Segment-F1 score of 0.857, compared to 0.690 for AudioComposer. In multi-event scenarios, its performance (Seg-F1-ME of 0.771) nearly doubled that of several competitors. Human evaluations further confirmed these findings, with PicoAudio2 receiving top marks for both audio quality (3.29 MOS) and temporal controllability (4.15 MOS).
Discussion and Conclusion
A key insight from this work is the importance of decoupling global semantics from local timing. This separation allows the model to handle complex, wordy prompts without losing track of the requested temporal structure. Furthermore, the inclusion of real-world data in the training process was found to be critical for maintaining the natural decay and onset of sounds, which purely synthetic models often fail to capture.
While PicoAudio2 sets a new standard for temporal control, it currently faces challenges with overlapping events due to limitations in the grounding models used for data curation. Future research will focus on enhancing the model's ability to generate complex, layered soundscapes where multiple events occur simultaneously.
Summary updated