Abstract
Detailed captions that accurately reflect the characteristics of a music piece can enrich music databases and drive forward research in music AI. This paper introduces a multi-task music captioning model, SonicVerse, that integrates caption generation with auxiliary music feature detection tasks such as key detection, vocals detection, and more, so as to directly capture both low-level acoustic details as well as high-level musical attributes. The key contribution is a projection-based architecture that transforms audio input into language tokens, while simultaneously detecting music features through dedicated auxiliary heads. The outputs of these heads are also projected into language tokens, to enhance the captioning input. This framework not only produces rich, descriptive captions for short music fragments but also directly enables the generation of detailed time-informed descriptions for longer music pieces, by chaining the outputs using a large-language model. To train the model, we extended the MusicBench dataset by annotating it with music features using MIRFLEX, a modular music feature extractor, resulting in paired audio, captions and music feature data. Experimental results show that incorporating features in this way improves the quality and detail of the generated captions.
Introduction
Automated music captioning is a complex cross-modal task. While existing models can often describe the general "vibe" or mood of a track, they frequently struggle with technical elements such as specific keys, chord progressions, or instrumentation. This limitation stems from two primary hurdles: the scarcity of large-scale datasets pairing high-quality audio with technically accurate descriptions, and the temporal complexity of full-length songs which evolve significantly over time.
SonicVerse addresses these challenges by explicitly predicting "hard" musical features—like key, instrumentation, and vocals—and injecting them as tokens into a Large Language Model (LLM).
Methodology
The SonicVerse architecture is modular, designed to bridge the gap between raw audio and natural language through a multi-task learning framework.
Overall Pipeline
The system processes audio in two stages. For short 10-second chunks, the SonicVerse model generates a detailed caption. For full-length pieces, these chunks are processed sequentially, and their individual captions are concatenated and synthesized by a higher-level LLM (GPT-4) to create a coherent, time-stamped narrative.
Multi-Task Projection Architecture
The core of SonicVerse lies in its projection mechanism. It utilizes a frozen MERT-95M encoder to extract hierarchical audio embeddings ($H$). These embeddings are then passed through two parallel pathways:
- Music Content Projector: Maps raw embeddings to content tokens ($z_{content}$) representing general acoustic characteristics.
- Music Feature Projector: Employs multiple auxiliary heads trained to classify specific attributes (key, genre, mood, etc.). These classifications are projected into feature tokens ($z_{feature}$).
These tokens, combined with a textual query, are fed into a frozen Mistral-7B LLM. By forcing the model to predict technical features and using those predictions as input tokens, the system gains a grounded understanding of the music's technical structure, reducing hallucinations.
Data Augmentation and Training
To overcome data scarcity, the researchers utilized MIRFLEX, a modular music feature extractor, to auto-annotate existing datasets including Jamendo (55k clips), Magna-Tag-A-Tune (25k clips), and MusicBench (26k clips). This augmentation provided the "hard" feature labels necessary for multi-task training. The model was evaluated against the standard MusicCaps set.
Results and Discussion
SonicVerse demonstrates significant improvements over baseline models and established competitors like MusiLingo and LLARK.
- Quantitative Performance: Ablation studies showed that the feature-augmented version outperformed control baselines across NLP metrics such as BLEU-4 (0.1824 vs 0.1799) and BERT Score (0.8723 vs 0.8716).
- Technical Accuracy: SonicVerse achieved a key detection accuracy of 0.3048, surpassing the much larger QWEN2-Audio (0.2966), which relies on significantly more private training data.
- Qualitative Depth: Case studies on complex tracks like Bohemian Rhapsody highlight the model's ability to track temporal transitions, identifying shifts in tempo, instrumentation, and vocal styles accurately.
Conclusion
SonicVerse presents a robust framework for music captioning by integrating explicit musical feature detection with LLM-based generation. While current limitations include a dependency on external LLMs for long-form synthesis and potential noise from auto-generated labels, the multi-task approach provides a clear path toward more technically accurate and temporally aware music AI. Future work will focus on native long-form synthesis and refining the granularity of feature detection.
Summary updated