Deep learning thrives on scale, but in the musical domain, data availability is highly asymmetrical. Unlabeled audio tracks, raw text corpora, and unaligned MIDI files are abundant. However, the exact data needed to train cross-modal models—such as syllable-level lyric-to-melody alignments, fine-grained video-to-beat synchronizations, or rich natural language descriptions of complex audio textures—is notoriously scarce. Copyright restrictions, the nuances of musical theory, and the immense cost of expert manual annotation create a severe bottleneck.
To build models that can generate, translate, or caption music, researchers cannot rely on massive parallel datasets. Instead, they must design architectures that can learn cross-modal mappings from unpaired data, intermediate abstractions, or frozen foundation models.
Intermediate Bridges and Templates
When direct mappings between Domain A (e.g., text) and Domain B (e.g., melody) are missing, a powerful strategy is to introduce an intermediate representation—a "bridge"—that can be derived from both sides independently. This decomposes a difficult, data-starved cross-modal task into two simpler tasks that can leverage abundant unaligned data.
[Text Description] ───(Supervised / Small Data)───> [Musical Attributes / Templates]
│
(Self-Supervised / Large Unlabeled Data)
▼
[Symbolic Music / MIDI]
For example, in text-to-symbolic-music generation, Peiling Lu et al. developed MuseCoco: Generating Symbolic Music from Text, which uses explicit musical attributes (like chord progressions, tempo, and instrumentation) as a bridge. Instead of mapping text directly to MIDI, the model maps text to attributes, and then attributes to MIDI. This allows the second stage to be trained entirely via self-supervision on massive unlabeled music datasets.
Similarly, in lyric-to-melody generation, Zeqian Ju et al. proposed TeleMelody: Lyric-to-Melody Generation with a Template-Based Two-Stage Method. The framework decomposes the pipeline into lyric-to-template and template-to-melody stages. By using a musical template containing structural and rhythmic constraints, the system achieves high data efficiency and controllability without requiring end-to-end paired data.
This concept of structural bridging also extends to multi-instrument arrangement. Hao-Min Liu et al. utilized "harmonic bridges" in Lead Sheet Generation and Arrangement by Conditional Generative Adversarial Network to learn how to arrange full accompaniments from basic lead sheets, even when professional side-by-side arrangements are unavailable.
Unsupervised Pre-training on Unpaired Corpora
Another core strategy is to learn the internal structure of each modality independently before attempting alignment. If a model deeply understands the syntax of lyrics and the geometry of melodies, it requires far fewer paired examples to learn how they fit together.
In SongMASS: Automatic Song Writing with Pre-training and Alignment Constraint, Zhonghao Sheng et al. utilized song-level masked pre-training on massive, completely unpaired datasets of lyrics and melodies. By training the model to recover masked sections within each modality separately, and then applying weak attention-based alignment constraints, the system achieves high-quality songwriting without large parallel corpora.
To capture the non-linear relationship between syllables and notes, Jiaxing Yu et al. introduced SongGLM: Lyric-to-Melody Generation with 2D Alignment Encoding and Multi-Task Pre-Training, which employs a 2D alignment encoding alongside a hierarchical multi-task pre-training strategy to maximize data efficiency.
Taking this a step further, Yufei Tian et al. developed Unsupervised Melody-to-Lyrics Generation (LYRA), an entirely unsupervised melody-to-lyrics framework. Instead of learning alignments from data, LYRA enforces music theory constraints directly during inference and uses multi-task learning for syllable control, completely eliminating the need for paired training data.
Aligning Frozen Foundation Models
Rather than training cross-modal representations from scratch, recent architectures leverage the pre-existing representations of large foundation models. If an audio encoder already understands music, and a large language model (LLM) already understands text, the problem reduces to aligning their latent spaces. This drastically reduces the sample complexity of the target task.
[Raw Audio] ──> [Frozen Music Encoder (MERT)] ──> [Projection Layer] ──> [Frozen LLM] ──> [Caption/Response]
In the domain of music captioning and question answering, Zihao Deng et al. introduced MusiLingo: Bridging Music and Text with Pre-trained Language Models for Music Captioning and Query Response, which connects a frozen music encoder (MERT) to a frozen LLM (Vicuna) via a lightweight projection layer. This approach requires significantly less paired data because the models do not need to learn musical or linguistic features from scratch—they only need to learn how to translate between them.
Similarly, Shansong Liu et al. developed Music Understanding LLaMA: Advancing Text-to-Music Generation with Question Answering and Captioning (MU-LLaMA) by combining MERT with LLaMA-2 to achieve advanced music comprehension despite the shortage of rich, natural language audio descriptions. This builds on earlier transfer learning insights from Ilaria Manco et al. in MusCaps: Generating Captions for Music Audio, who demonstrated that using pre-trained music auto-tagging models as feature extractors is key to overcoming data scarcity in audio captioning.
Retrieval-Augmentation and Constrained Inference
When generative models lack enough paired data to generalize smoothly, we can inject external knowledge or explicit rules during inference to guide the generation process, bypassing the need for end-to-end learned mappings.
For music captioning, Nikita Srivatsan et al. proposed Retrieval Guided Music Captioning via Multimodal Prefixes (RGMC), a hybrid system that retrieves a "neighbor" caption from a small dataset and feeds it alongside audio features as a "multimodal prefix" into a language model. This retrieval step provides a strong semantic anchor, allowing high-quality caption generation without a massive training corpus.
For lyric-to-melody generation, Ang Lv et al. introduced Re-creation of Creations: A New Paradigm for Lyric-to-Melody Generation (ROC), which pairs a melody language model with a retrieval-based pipeline guided by human composition rules. By offloading the structural alignment to explicit rules and retrieved exemplars, the system sidesteps the need for end-to-end parallel data.
This paradigm of rule-based guidance is also vital for cross-lingual tasks. Fenfei Guo et al. developed Automatic Song Translation for Tonal Languages (GagaST) for automatic song translation into tonal languages, using constrained decoding to force translated lyrics to match the original melody's pitch contour and rhythm without relying on parallel translation datasets.
Key Foundational Work
SongMASS: Automatic Song Writing with Pre-training and Alignment Constraint (2021) — contributed 1 concept elements to this direction.
MusCaps: Generating Captions for Music Audio (2021) — contributed 1 concept elements to this direction.
A Melody-Conditioned Lyrics Language Model (2018) — contributed 1 concept elements to this direction.