Abstract
Automated audio captioning (AAC) is a multimodal task that involves generating natural language descriptions for audio signals. This paper introduces a robust AAC system utilizing a 10-layer CNN encoder and a Transformer decoder. By implementing a three-stage training strategy—pre-training on a custom multi-label classification task, training with advanced regularization, and fine-tuning—the proposed model achieves a state-of-the-art SPIDEr score of 0.227 on the DCASE 2020 Task 6 challenge.
Introduction
Automated audio captioning goes beyond simple audio tagging. While tagging identifies discrete events like "dog barking," AAC must capture spatiotemporal relationships (e.g., "a dog barks followed by a car driving by") and nuanced physical properties. The primary obstacle in this field is the scarcity of annotated data; training deep neural networks to map complex acoustic features to fluent English requires significant data to avoid overfitting and capture acoustic nuances.
Methodology
The authors propose a sequence-to-sequence (seq2seq) architecture characterized by a three-stage training pipeline designed to maximize performance on limited data.
1. Encoder-Decoder Architecture
- Encoder (CNN10): A 10-layer Convolutional Neural Network extracts high-level features from log Mel-spectrograms.
- Decoder (Transformer): Unlike traditional LSTM-based decoders, a 2-layer Transformer is employed to better handle long-range dependencies and map acoustic features to natural language.
2. Three-Stage Training Strategy
To overcome data shortages, the model follows a specific progression:
- Pre-training: The CNN encoder is first trained on a multi-label classification task. The target classes (e.g., "chirp," "talk") are derived from the most frequent words in the captioning dataset itself.
- Training: The encoder parameters are frozen, and the Transformer decoder is trained using the extracted feature sequences.
- Fine-tuning: The entire model is fine-tuned with a low learning rate to optimize end-to-end performance.
3. Regularization Techniques
Two key techniques were used to improve generalization:
- SpecAugment: Randomly masking frequency and time blocks in the spectrogram to increase model robustness.
- Label Smoothing: Preventing over-confidence in word predictions to enhance the model's ability to generalize to unseen data.
Experimental Setup
The study utilizes the Clotho dataset, the benchmark for the DCASE 2020 Task 6 challenge. It consists of 4,981 audio clips, each 15–30 seconds long, with five unique captions per clip. The vocabulary comprises 4,368 unique words.
Results and Discussion
The model was evaluated using standard metrics, with SPIDEr (the average of CIDEr and SPICE) as the primary benchmark. The results demonstrate that the combination of a Transformer architecture and pre-training significantly outperforms baseline models.
Key insights from the results include:
- Pre-training is Essential: Forcing the encoder to learn "tagging" first creates a superior latent representation of the audio.
- Transformer Superiority: The Transformer's self-attention mechanism proves more effective than LSTMs for establishing mappings between acoustic events and fluent text.
- Ablation Impact: As shown in the ablation study, while pre-training provides the foundation, the addition of SpecAugment (SA) and Label Smoothing (LS) is critical for reaching top-tier performance.
Conclusion
This research highlights the efficacy of combining CNN-based feature extraction with Transformer-based language modeling for audio captioning. Future work may explore larger-scale pre-training on datasets like AudioSet and more complex attention mechanisms to better align specific audio timestamps with generated words.
Summary updated