Abstract
Audio captioning involves generating descriptive natural language for acoustic environments. This work introduces a lightweight (5M parameter) framework consisting of a 5-layer CRNN encoder and a GRU decoder. By implementing a two-stage training process—standard cross-entropy followed by reinforcement learning via Self-Critical Sequence Training (SCST)—the authors mitigate exposure bias and align the model's objective with human-centric evaluation metrics. The approach demonstrated top-tier performance in the DCASE 2020 challenge, ranking 4th overall while maintaining high computational efficiency.
Introduction
Unlike sound event detection, which identifies discrete labels, audio captioning requires a holistic understanding of temporal and spectral relationships to produce fluent descriptions. Current research faces two primary obstacles: exposure bias, where models fail to recover from their own errors during inference because they were trained only on ground-truth tokens, and objective mismatch, where minimizing cross-entropy loss does not necessarily maximize evaluation metrics like CIDEr or BLEU.
System Architecture
The proposed model utilizes an encoder-decoder structure designed for both efficiency and representational power, totaling only 5 million parameters.
Encoder: 5-Layer CRNN
The encoder processes log-Mel spectrograms through five convolutional blocks to extract local frequency features. These are followed by a Bidirectional GRU (BiGRU) layer to capture temporal dependencies across the audio clip. A Global Average Pooling (GAP) layer then compresses the output into a fixed 256-dimensional audio embedding.
Decoder: GRU
The decoder is a single-layer GRU with 512 units. It generates captions word-by-word by attending to the audio embedding and the previously generated token, as shown in the architecture overview below.
Training Methodology
The training is conducted in two distinct stages to bridge the gap between language modeling and evaluation metrics:
- Cross-Entropy (XE) Training: The model first learns basic language structure and audio-text alignment using standard teacher forcing.
- Reinforcement Learning (RL) Fine-tuning: Using Self-Critical Sequence Training (SCST), the model generates sequences and receives a "reward" based on the CIDEr score. This stage forces the model to rely on its own predictions during training, effectively reducing exposure bias and optimizing directly for the metrics used in competition.
Experimental Results
Evaluated on the Clotho dataset (4,981 audio samples), the CRNN-based models significantly outperformed GRU-only baselines. The inclusion of reinforcement learning and data augmentation proved critical for high performance.
Key findings from the experimental results include:
- Impact of RL: Fine-tuning with SCST (CRNN-R) improved the SPIDEr score from 0.142 to 0.194 compared to the base CRNN model.
- Data Augmentation: Incorporating SpecAugment (CRNN-R-SpecAug) further boosted the SPIDEr score to 0.223, outperforming many complex ensemble models.
- Efficiency: The system achieves competitive results with only 5 million parameters, making it highly suitable for resource-constrained environments compared to other DCASE entries using over 30 million parameters.
Conclusion
The combination of a CRNN encoder and RL-based fine-tuning provides a robust and efficient solution for audio captioning. While the model occasionally favors general vocabulary (e.g., "running") over highly specific human descriptors (e.g., "vibrating"), its ability to optimize directly for evaluation metrics makes it a powerful and lightweight architecture for automated audio description.
Summary updated