CSL-L2M: Controllable Song-Level Lyric-to-Melody Generation Based on Conditional Transformer with Fine-Grained Lyric and Musical Controls

Li Chai, Donglin WangAAAI 2025 · DOI

TL;DR

This paper introduces CSL-L2M, a novel method for generating full-song melodies from lyrics with fine-grained control over both lyrical and musical attributes. By developing a new REMI-Aligned representation and a multi-granularity control mechanism, CSL-L2M significantly improves melody quality, controllability, and structural coherence compared to existing approaches.

What Problem Does This Paper Address?

Generating music that aligns with lyrics is a fundamental task in AI music generation, but it's notoriously challenging. Previous methods often struggle with:

  • Weak Correlations: The relationship between lyrics and melodies is complex and not always a direct one-to-one mapping, making it hard for models to learn.
  • Poor Controllability: Users have limited ability to influence the generated melody's style, mood, or specific musical characteristics.
  • Low Quality and Structure: Generated melodies can sound uninspired, lack coherence, or fail to exhibit typical song structures like verses and choruses, especially for full-length pieces.
  • Alignment Issues: Accurately aligning each syllable of the lyrics to the correct notes in the melody is difficult, with "one-to-many" alignments being common but hard to model.

This paper aims to tackle these issues by enabling controllable, high-quality, and structurally sound full-song melody generation from lyrics.

Methods

CSL-L2M employs a conditional Transformer decoder enhanced with an "in-attention" mechanism for fine-grained control. Key innovations include:

  • REMI-Aligned Representation: A new music representation that explicitly incorporates syllable- and sentence-level alignments between lyrics and melodies. This allows for more precise modeling of the lyrical-musical relationship, including "one-to-many" alignments.
  • Multi-Granularity Lyric Controls:
    • Syllable-level Tone Embeddings: Captures tonal nuances of lyrics, crucial for tonal languages like Chinese, to guide melody pitch.
  • Word-level Part-of-Speech (POS) Embeddings: Helps in modeling prosodic boundaries and enhancing rhythm and structure.
  • Sentence-level Semantic Embeddings: Extracted by a bidirectional Transformer encoder, these capture the overall meaning of lyrical sentences.
  • Multi-Granularity Musical Controls:
    • Coarse-grained: Human-labeled musical tags (key, emotion, structure) provide high-level stylistic guidance.
  • Fine-grained: Sentence-level statistical musical attributes (e.g., pitch mean/variance, duration mean/variance) offer more detailed control over musical characteristics.
  • High-fidelity: Learned musical features from a pre-trained VQ-VAE model provide rich, detailed musical information.
  • In-Attention Mechanism: This technique allows the Transformer decoder to tightly integrate the various lyric and musical control signals at each generation step, ensuring fine-grained influence over the output melody.

Datasets

The primary dataset used is a newly curated, high-quality, precisely annotated parallel lyric-melody dataset consisting of 10,170 Chinese pop songs. This dataset includes:

  • Paired lyrics and melodies with precise synchronization.
  • Musical tags: key, emotion, and song structure (verse, chorus, etc.).
  • Beats per minute (BPM).

The dataset is split into training, validation, and test sets in a 9:0.5:0.5 ratio.

Key Insights and Ideas

  • Explicit Alignment is Crucial: The REMI-Aligned representation highlights the importance of explicitly modeling lyric-melody alignment at multiple levels (syllable, sentence) for better generation.
  • Multi-Granularity Controls Unlock Flexibility: Combining coarse (tags) and fine-grained (statistical attributes, learned features) controls for both lyrics and music provides a powerful way to guide melody generation.
  • In-Attention for Tight Control: The in-attention mechanism is key to effectively leveraging these diverse control signals within the Transformer decoder, preventing control signals from being "forgotten" during long sequence generation.
  • Lyric Information Beyond Semantics: Incorporating tonal and POS information from lyrics provides richer cues for melody generation than just semantic meaning alone.
  • Musical Controls Aid Melody Modeling: Musical controls not only enable user-driven generation but also provide valuable musical context that helps the model learn better melody patterns.

Key Results

  • Objective Performance: CSL-L2M significantly outperforms state-of-the-art models (SongComposer, TeleMelody) in objective metrics like Pitch Distribution Similarity (PD) and Duration Distribution Similarity (DD), achieving 86.35% and 93.50% respectively, with a Melody Distance (MD) of 1.27.
  • Subjective Performance: Human evaluations show CSL-L2M generates melodies with higher quality, better harmony, rhythm, structure, emotion, and overall quality compared to baselines.
  • Controllability: Ablation studies demonstrate that removing different control mechanisms (duration, pitch, note density, alignment, musical controls) leads to a continuous degradation in performance, validating the effectiveness of each control type.
  • High-Fidelity Generation: The version of CSL-L2M incorporating learned musical features achieves near-ceiling performance, indicating the value of high-fidelity musical information.
  • Independent Controllability: Spearman's rank correlation analysis confirms that the statistical musical controls are largely independent, allowing users to adjust one attribute without drastically affecting others.

Limitations and Future Work

  • Learned Features Dependency: The learned musical features require pre-trained models and existing melodies, which might limit their applicability in scenarios where no such prior data is available.
  • Dataset Specificity: While the dataset is large, it's specific to Chinese pop songs. Future work could explore generalization to other genres or languages.
  • Future Directions: The authors suggest exploring style transfer and more advanced controllable generation scenarios, building upon the foundation laid by CSL-L2M. The paper also implies that further research into the interplay between different control granularities could yield even better results.

Summary updated

BibTeX
@inproceedings{chai2025csl,
  author = {Li Chai and Donglin Wang},
  title = {{CSL-L2M: Controllable Song-Level Lyric-to-Melody Generation Based on Conditional Transformer with Fine-Grained Lyric and Musical Controls}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i22.34523},
}
Figure