A Detailed Audio-Text Data Simulation Pipeline Using Single-Event Sounds

Xuenan Xu, Xiaohang Xu, Zeyu Xie, Pingyue Zhang, Mengyue Wu, Kai Yu

Abstract

Recent advancements in multimodal AI have seen significant progress in image-to-text tasks, yet audio captioning continues to struggle with a lack of high-quality, granular data. Existing datasets often rely on human annotations that are subjective and lack temporal or quantitative precision. This paper presents an automated pipeline designed to generate "rich-detail" audio-text pairs by synthesizing complex acoustic scenes from single-event sounds. By leveraging Large Language Models (LLMs) to translate simulation parameters into natural language, the pipeline produces ground-truth captions that accurately reflect event counts, timing, and spatial relationships. Experimental results demonstrate that models fine-tuned on this simulated data achieve a nearly two-fold increase in temporal accuracy and are significantly preferred by human evaluators.

Introduction

In the field of audio-language modeling, the quality of training data remains a primary bottleneck. While datasets like AudioCaps and Clotho have provided a foundation for the field, their captions are often high-level and omit the nuances of acoustic scenes. For instance, a human annotator might describe a recording as "a dog barking," failing to note that the dog barked exactly three times or that the barking occurred against a background of distant traffic.

Human annotation is inherently subjective and prone to error when dealing with overlapping sounds or rapid sequences. To address this, we propose a shift from manual annotation to a controlled simulation framework. By building audio scenes from the ground up, we can generate "perfect" metadata—a ground-truth recipe of every event in the audio—and use that metadata to produce highly descriptive, accurate captions.

The Three-Stage Simulation Pipeline

The proposed pipeline consists of three distinct phases: taxonomy development, single-event curation, and automated synthesis.

1. Auditory Detail Taxonomy

To understand what details are most valuable, the researchers analyzed the AudioCaps corpus to identify recurring descriptive elements. This resulted in a taxonomy of 10 specific auditory aspects, including:

  • Identity: The specific sounding objects.
  • Occurrence: The number of times an event happens.
  • Temporal Relationships: The order and overlap of sounds.
  • Loudness and Spatiality: The volume and perceived environment of the sound.

2. Single-Event Curation

High-quality simulation requires clean, isolated building blocks. The authors sourced raw audio from the Freesound platform, employing a multi-stage filtering process to ensure quality:

  • TAG Model: Used to identify specific target segments within longer clips.
  • CLAP Model: A Contrastive Language-Audio Pretraining model was used to verify that the audio content matched its textual description.
  • Acoustic Clustering: To resolve ambiguities in library tags (e.g., distinguishing between "liquid" and "water"), the authors used CLAP embeddings and K-means clustering to group sounds based on their actual acoustic signatures.

3. Simulation and Captioning

The final stage involves mixing these curated single events into complex scenes. Unlike traditional data augmentation, this process is parameter-driven. The system randomly samples events and mixes them in the time domain, recording the exact "recipe" (e.g., "Sound A occurs at 1.0s, Sound B starts at 2.5s at 50% volume").

This structured metadata is then fed into an LLM (ChatGPT). The LLM acts as a translator, converting the technical simulation parameters into a natural, fluent caption. This ensures that the resulting text is not just a list of labels, but a descriptive narrative that accurately reflects the temporal and quantitative reality of the audio.

Experimental Results

The effectiveness of the pipeline was tested by producing a curated set of 2,785 rich-detail audio-text pairs. A baseline audio captioning model was fine-tuned on this simulated dataset, leading to several key findings:

Quantitative Improvements

The fine-tuned model showed consistent gains across standard NLP metrics such as BLEU-4, ROUGE, and METEOR. The most significant improvement was observed in the Temporal $F_1$ score, which measures the accuracy of event sequencing in captions. This score rose from 35.1 to 60.9, indicating a vastly improved ability to perceive and describe the timing of sounds.

Human Evaluation

In blind head-to-head tests, human raters were asked to compare captions from the baseline model and the fine-tuned model. The results were decisive:

  • Accuracy: Raters preferred the fine-tuned model’s captions 62% of the time.
  • Detailedness: The fine-tuned model was preferred 82% of the time.

Qualitative analysis showed that the model transitioned from vague descriptions (e.g., "a cat meows") to precise ones (e.g., "a cat meows twice"), demonstrating a newfound ability to count and sequence acoustic events.

Discussion and Limitations

While the simulation pipeline offers a powerful new tool for data generation, certain challenges remain. The current dataset size is relatively small, and scaling to hundreds of thousands of samples is necessary for broader applications. Additionally, the pipeline currently focuses on "hard" data like counts and timing. "Soft" details, such as the emotional quality of a sound (e.g., a "mournful" violin), are more difficult to simulate and require further research into emotion-classification models.

Finally, there is the risk of a distribution shift. Simulated audio is often "cleaner" than real-world recordings. Future iterations of the pipeline will focus on introducing complex background textures and environmental noise to better mimic the messiness of field recordings.

Conclusion

By moving from subjective human annotation to a parameter-driven simulation, this research provides a scalable path toward more detailed and accurate audio-text datasets. The significant jump in temporal accuracy suggests that when models are given precise ground-truth data, they are capable of much more sophisticated acoustic perception than previously demonstrated.

Summary updated

BibTeX
@inproceedings{xu2024detailed,
  author = {Xuenan Xu and Xiaohang Xu and Zeyu Xie and Pingyue Zhang and Mengyue Wu and Kai Yu},
  title = {{A Detailed Audio-Text Data Simulation Pipeline Using Single-Event Sounds}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2024},
  url = {https://openreview.net/forum?id=DjSCcXJi80},
}
Figure