SonicVerse: Multi-Task Learning for Music Feature-Informed Captioning

Anuradha Chopra, Abhinaba Roy, Dorien Herremans

TL;DR

SonicVerse is a novel multi-task learning framework that enhances music captioning by explicitly predicting "hard" musical features—like key, instrumentation, and vocals—and injecting them as tokens into a Large Language Model (LLM). By combining these technical features with general audio embeddings and a unique "LLM-chaining" strategy, the system generates highly detailed, temporally-aware descriptions for both short clips and full-length songs.

What Problem Does This Paper Address?

Automated music captioning is a difficult "cross-modal" task. While existing models can describe the "vibe" or mood of a track (soft features), they often struggle to identify technical elements like the specific key, chord progressions, or instrumentation (hard features).

The field faces two major hurdles:

  1. Data Scarcity: There are very few large-scale datasets that pair high-quality audio with detailed, technically accurate text descriptions.
  2. Temporal Complexity: Most models are trained on short (10–30 second) clips. Describing how a 5-minute song evolves over time—transitioning from a verse to a chorus or changing instrumentation—remains an open challenge.

Methods

SonicVerse utilizes a modular architecture built on three main components:

  • The Encoder: It uses a frozen MERT-95M model, a SOTA music understanding model, to extract hierarchical audio embeddings.
  • The Multi-Task Projector: This is the "secret sauce." Instead of a simple linear mapping to text, the authors use two parallel pathways:
  • Music Content Projector: Maps raw embeddings to "content tokens" ($z_{content}$) representing general acoustic characteristics.
  • Music Feature Projector: Features multiple "auxiliary heads" trained to classify specific attributes (key, genre, mood, etc.). The outputs of these heads are then projected into "feature tokens" ($z_{feature}$).
  • The LLM: These tokens, along with a textual query, are fed into a frozen Mistral-7B model to generate the final natural language caption.

To handle long-form music, the authors implement LLM-enabled chaining. The system clips a long song into 10-second chunks, captions each individually, and then feeds the concatenated results into GPT-4 with a specific prompt to synthesize a coherent, time-stamped narrative of the entire piece.

Datasets

The researchers utilized three primary datasets, but with a significant twist:

  • Jamendo (55k clips), Magna-Tag-A-Tune (25k clips), and MusicBench (26k clips).
  • The Augmentation: Since these datasets lacked "hard" feature labels, the authors used MIRFLEX (a modular music feature extractor) to auto-annotate them with labels for key, instrumentation, and vocals. This allowed for multi-task training even on datasets that weren't originally designed for it.
  • Evaluation: Models were tested on the standard MusicCaps evaluation set.

Key Insights and Ideas

  • Features as Guidance: The core "aha moment" is that explicitly forcing the model to predict technical features (and using those predictions as input tokens) acts as a powerful guide for the LLM. It prevents the model from hallucinating "vibes" that don't match the actual music theory of the track.
  • Auxiliary Supervision: Multi-task learning allows the model to learn effectively from smaller, open-source datasets. The auxiliary classification tasks provide a "stronger" signal than just trying to predict the next word in a sentence.
  • Bypassing the Context Window: By using GPT-4 to "summarize" a chain of short-clip captions, the authors solve the problem of audio encoders having limited temporal resolution without needing to train on massive, long-form audio files.

Key Results

  • Ablation Success: In internal tests, the "Feature Projector Augmented" version (SonicVerse) outperformed the control baseline across nearly all NLP metrics, including BLEU-4 (0.1824 vs 0.1799) and BERT Score (0.8723 vs 0.8716).
  • SOTA Comparison: SonicVerse outperformed several established models like MusiLingo and LLARK. Notably, it achieved higher Key Detection accuracy (0.3048) than the much larger QWEN2-Audio (0.2966), despite QWEN2 being trained on significantly more private data.
  • Qualitative Depth: The case study on Queen’s Bohemian Rhapsody demonstrated the model's ability to track complex transitions, identifying the shift from a "gospel-inspired section" to a "slower tempo" with a male vocalist and cello.

Limitations and Future Work

  • Noisy Labels: The model was trained on labels generated by MIRFLEX, not human experts. If the feature extractor makes a mistake, that error can propagate into the training of the captioning model.
  • External LLM Dependency: The long-form chaining currently relies on GPT-4, which is a closed-source, paid API. The authors suggest that future work could involve fine-tuning the internal LLM (Mistral) to handle this synthesis natively.
  • Temporal Resolution: While 10-second chunks work well, some musical events happen much faster. Increasing the granularity of the feature heads could lead to even more precise time-stamped captions.

Summary updated

Abstract

Detailed captions that accurately reflect the characteristics of a music piece can enrich music databases and drive forward research in music AI. This paper introduces a multi-task music captioning model, SonicVerse, that integrates caption generation with auxiliary music feature detection tasks such as key detection, vocals detection, and more, so as to directly capture both low-level acoustic details as well as high-level musical attributes. The key contribution is a projection-based architecture that transforms audio input into language tokens, while simultaneously detecting music features through dedicated auxiliary heads. The outputs of these heads are also projected into language tokens, to enhance the captioning input. This framework not only produces rich, descriptive captions for short music fragments but also directly enables the generation of detailed time-informed descriptions for longer music pieces, by chaining the outputs using a large-language model. To train the model, we extended the MusicBench dataset by annotating it with music features using MIRFLEX, a modular music feature extractor, resulting in paired audio, captions and music feature data. Experimental results show that incorporating features in this way improves the quality and detail of the generated captions.
BibTeX
@inproceedings{chopra2025sonicverse,
  author = {Anuradha Chopra and Abhinaba Roy and Dorien Herremans},
  title = {{SonicVerse: Multi-Task Learning for Music Feature-Informed Captioning}},
  booktitle = {Aimc},
  year = {2025},
  doi = {10.5281/zenodo.16955817},
}

Loading related content...

Figure