TL;DR
This paper introduces EnCLAP, a novel framework for automated audio captioning that combines a neural audio codec (EnCodec) with a joint audio-text embedding model (CLAP) and a pre-trained language model (BART). They also propose a new training objective, masked codec modeling, to enhance the language model's understanding of acoustic nuances, achieving state-of-the-art results on standard audio captioning benchmarks.
What Problem Does This Paper Address?
Automated Audio Captioning (AAC) aims to generate natural language descriptions for audio clips. While encoder-decoder architectures are common, a significant performance gap remains between automated systems and human capabilities. This is partly due to the inherent complexity of capturing diverse sound events and the relative scarcity of labeled audio data compared to image captioning datasets. Existing approaches often rely on transfer learning, using pre-trained audio encoders and language models, but there's room for improvement in how acoustic information is represented and integrated into the captioning process.
Methods
EnCLAP tackles this by employing a multi-pronged approach:
- Dual Acoustic Representations: Instead of relying on a single audio encoder, EnCLAP uses two:
- CLAP (Contrastive Language-Audio Pretraining): This model provides a sequence-level embedding that captures the overall semantic content of the audio.
- EnCodec: This neural audio codec generates discrete "code" sequences. The authors hypothesize that discrete inputs are more amenable to pre-trained language models like BART, and they use these codes as time-step level acoustic features.
- Pre-trained Language Model (BART): A pre-trained BART model serves as the caption decoder. The inputs from CLAP and EnCodec are combined and fed into BART.
- Masked Codec Modeling (MCM): A novel auxiliary training objective is introduced. This task involves masking portions of the EnCodec's discrete code sequences and training BART to predict the original codes. This is designed to improve BART's "acoustic awareness" by forcing it to learn finer-grained acoustic relationships.
- Combined Input for BART: The CLAP embedding is projected and concatenated with the BART positional embeddings of the EnCodec codes (after they are embedded by BART's own embedding layers) to form the final input to BART.
Datasets
The framework is trained and evaluated on two widely used audio captioning datasets:
- AudioCaps: Contains approximately 50K training captions.
- Clotho: Contains around 3839 training examples (version 2.1), with each audio clip having 5 captions.
Key Insights and Ideas
- Discrete Codes are Powerful: The paper strongly suggests that discrete acoustic codes from neural codecs like EnCodec, when combined with sequence-level embeddings (from CLAP), provide a more effective input to pre-trained language models than traditional continuous representations like log-mel spectrograms.
- Auxiliary Tasks Boost Acoustic Awareness: The proposed Masked Codec Modeling (MCM) task significantly improves the language model's ability to understand acoustic details, leading to better captioning performance.
- Synergy of CLAP and EnCodec: Combining a semantic audio embedding (CLAP) with discrete, time-step level acoustic features (EnCodec) yields superior results compared to using either alone or with other acoustic representations.
Key Results
EnCLAP achieves state-of-the-art performance on the AudioCaps dataset. Specifically, the "EnCLAP-large" model sets a new benchmark, outperforming even models pre-trained on much larger datasets. On the Clotho dataset, EnCLAP models also consistently outperform baseline methods. Ablation studies confirm that each component of EnCLAP (CLAP, EnCodec, MCM training) contributes to the overall performance, with CLAP audio embedding providing the most significant improvement.
Limitations and Future Work
The authors note that the "EnCLAP-large" model shows inconsistent performance on Clotho when trained solely on it, suggesting it might be more prone to overfitting in low-resource settings. They hypothesize that more careful regularization might be needed.
Future work includes expanding EnCLAP's application to other tasks such as music captioning and audio generation.
Summary updated