Abstract
SILA (Signal-to-Language Augmentation) is a model-agnostic framework designed to provide fine-grained control over acoustic properties—such as loudness, pitch, and reverb—in text-to-audio generation. By automatically extracting these features from audio and appending them as text descriptors to training captions, the framework enables models to disentangle a sound's identity from its specific acoustic characteristics.
Introduction
Current Text-to-Audio (TTA) models are highly effective at generating sounds from semantic descriptions (e.g., "a dog barking"). However, they often fail to provide "fine-grained control" over specific acoustic nuances. For instance, requesting a "quiet, high-pitched dog bark with heavy reverb" typically results in a generic bark. This limitation stems from text encoder constraints, a lack of detailed acoustic labels in training data, and the entanglement of sound identity with common acoustic profiles (e.g., explosions being inherently loud in datasets).
Methodology
The SILA framework addresses these challenges through a three-stage "Signal-to-Language" pipeline that transforms raw audio signals into descriptive text tokens for training.
- Coarse Caption Generation: An audio language model (GAMA) generates base captions, which are refined by Mistral-7B using metadata to create detailed descriptions.
- DSP-Based Feature Extraction: Digital Signal Processing (DSP) and deep learning extract seven key descriptors: loudness, pitch (via CREPE), reverb (synthesized via Pedalboard), noise (SNR), brightness (spectral centroid), fade, and duration.
- Augmented Training: These descriptors are appended to text prompts (e.g., "Explosion, & loudness: soft"). A Diffusion Transformer (DiT) conditioned on FLAN-T5 XXL embeddings learns to associate these specific tokens with corresponding signal variations.
Evaluation and Results
The model was trained on a mix of FreeSound and proprietary datasets and evaluated zero-shot on the Adobe AuditionSFX dataset. SILA was compared against established models including Stable Audio Open, AudioGen, and Tango 2.
Objective Performance
SILA achieved a state-of-the-art CLAP score of 0.29, indicating superior alignment between text prompts and generated audio. It also maintained high audio quality with a competitive FAD score of 0.84, demonstrating that increased control does not compromise sound plausibility.
Subjective Control
User studies revealed a strong preference for SILA in specific control tasks. As shown in the evaluation results, SILA significantly outperformed baselines across all acoustic categories.
For example, in loudness control, users preferred SILA 70% of the time, compared to only 17% for the next best baseline, Stable Audio Open. This confirms that explicitly labeling acoustic features allows the model to treat them as independent axes of variation.
Conclusion and Future Work
SILA demonstrates that fine-grained control in TTA can be achieved through data-preprocessing and prompting strategies without complex architectural changes. Future work will focus on handling compositional audio scenes with overlapping events, integrating descriptors into more natural language phrasing, and expanding the framework to text-to-speech and music generation domains.
Summary updated