S²MILE: Semantic-and-Structure-Aware Music-Driven Lyric Generation

Mu You, Fang Zhang, Shuai Zhang, Linli XuAAAI 2025 · DOI

Abstract

Generating lyrics that align with both the emotional nuances and the rhythmic structure of polyphonic music remains a significant challenge in AI music composition. Existing models often focus solely on monophonic melodies or rely on rigid note-to-syllable mappings, leading to semantically mismatched or rhythmically awkward results. We present S²MILE, an end-to-end framework designed to bridge these "semantic" and "structural" gaps. By employing a hierarchical music information extractor and a dedicated syllable-length predictor, S²MILE generates lyrics that are both thematically resonant and rhythmically precise.

Introduction

Most current music-to-lyric systems oversimplify the creative process by focusing exclusively on the melody. This approach ignores the rich contextual information provided by the full arrangement—such as drums, bass, and chords—which defines the "vibe" or genre of a track. Furthermore, these systems often enforce a strict one-to-one mapping between musical notes and syllables, failing to capture the fluid phrasing characteristic of human lyricists.

S²MILE (Semantic-and-Structure-Aware Music-Driven Lyric Generation) addresses these limitations by replicating the human approach to songwriting: identifying high-level themes from the full musical arrangement and fitting words into rhythmic pockets. By moving beyond monophonic melodies to multi-instrumental data, the model gains a much better understanding of the song's genre and emotional weight.

Methodology

The S²MILE architecture consists of three primary components designed to process complex, multi-instrumental compositions.

The S²MILE architecture overview, featuring the hierarchical extractor, length predictor, and LLM-based generator.

1. Hierarchical Music Information Extractor

The framework extracts musical features at two distinct levels:

  • Song-level: An HTS-AT audio encoder paired with a Querying Transformer (Q-Former) captures global semantics, such as emotional tone and thematic elements. This "summary" provides the necessary mood for the lyrics.
  • Sentence-level: A T5-based encoder processes specific note sequences for individual lines. To accommodate polyphony (multiple instruments playing at once), the model utilizes a "zigzag-order" pattern, sequencing notes chronologically while maintaining instrument and pitch identity.

2. Lyric Length Predictor

Rather than adhering to a rigid note-syllable ratio, this module analyzes musical "energy"—specifically pitch variance and velocity centroid—to predict the optimal syllable count for a given phrase. This acts as a crucial middle ground between rigid rules and total chaos, leading to better rhythmic alignment.

3. Lyric Generator

The backbone of the system is Mistral-7B, a powerful Large Language Model (LLM). The authors use Low-Rank Adaptation (LoRA) to fine-tune the model, injecting the extracted musical features and predicted lengths into the prompt to guide the autoregressive generation of lyrics.

Technical Innovation: Polyphonic Representation

A key challenge in processing multi-instrumental music is representing simultaneous notes in a linear sequence suitable for Transformers. S²MILE introduces a "zigzag" sorting method, where notes occurring at the same timestamp are ordered by instrument ID and then by descending pitch.

The sentence-level encoder and the zigzag-order note sequencing used to handle polyphonic musical data.

To further refine the extractor, the authors implemented an Instrument Contrastive Loss. This encourages the model to distinguish between the unique sonic signatures of different instruments during pre-training, enhancing its ability to interpret complex arrangements.

Experiments and Results

The model was trained on a high-quality subset of the MetaMIDI dataset, comprising 53,367 tracks and over 1 million note-sequence/lyric pairs.

Quantitative Performance

S²MILE significantly outperformed baselines such as SongMASS, GPT-3.5, and SongComposer across several metrics:

  • Semantic Alignment: The model achieved superior CLAP (0.057) and CLaMP (0.203) scores, indicating that the generated lyrics are more thematically consistent with the audio than those produced by general-purpose LLMs.
  • Structural Precision: S²MILE demonstrated a lower Syllable Discrepancy Distance (SDD), confirming that its lyrics align more accurately with the musical timing.

Human Evaluation

In subjective testing, S²MILE received the highest ratings for Meaningfulness (3.75/5) and Coherence (3.60/5), approaching the quality of human-written ground truth.

Conclusion

S²MILE represents a significant step forward in automated lyric generation by moving beyond simple melodies to embrace the complexity of full musical arrangements. While the current model excels at line-by-line generation, it currently lacks a mechanism to ensure long-term narrative consistency across an entire song. Future work will focus on maintaining long-range context to manage verse-chorus structures and story arcs more effectively.

Summary updated

BibTeX
@inproceedings{you2025s,
  author = {Mu You and Fang Zhang and Shuai Zhang and Linli Xu},
  title = {{S²MILE: Semantic-and-Structure-Aware Music-Driven Lyric Generation}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i21.34375},
}
Figure