Song Lyrics Adaptations: Computational Interpretation of the Pentathlon Principle

Barbora Štěpánková, Rudolf RosaNAACL 2025

TL;DR

This paper tackles the challenge of computationally evaluating and generating singable song lyric translations. They propose a novel system that interprets and operationalizes Low's Pentathlon Principle, breaking it down into five measurable metrics, and demonstrate its effectiveness on the English-Czech language pair.

What Problem Does This Paper Address?

Translating song lyrics is notoriously difficult because it requires balancing multiple, often conflicting, objectives: the translation must be semantically accurate to the original, sound natural in the target language, maintain a singable rhythm and rhyme scheme, and ideally, be adaptable to the original melody. Traditional machine translation often fails to capture these nuances, leading to lyrics that are either nonsensical or unsingable. While Low's Pentathlon Principle offers a framework for evaluating singable translations, it has largely remained a qualitative guideline. This paper aims to bridge this gap by computationally interpreting and verifying the Pentathlon Principle, and then using this interpretation to build an automated system for generating singable lyric adaptations.

Methods

The core of this work lies in computationally defining and measuring the five aspects of Low's Pentathlon Principle:

  • Singability: Measured by the Consonant Cluster and Vowel Openness Distance (CCVO Dist). This metric analyzes phonetic transcriptions to quantify how easily consonant clusters and vowel openness in the translated lyrics would fit a melody, aiming for lower distances.
  • Sense: Assessed using Semantic Similarity based on sentence embeddings from a pre-trained Sentence BERT model, aiming for higher similarity.
  • Naturalness: Quantified by the perplexity of a language model trained on the target language (Czech in this case). Lower perplexity indicates more natural phrasing and word order.
  • Rhyme: Evaluated using the Rhyme Scheme Jaccard Index, which compares the rhyme schemes of the original and translated lyrics, aiming for a balance between retaining the original scheme and allowing for adaptations.
  • Rhythm: Measured by Syllable Distance, which calculates the difference in syllable counts between corresponding lines in the original and translated lyrics, aiming for lower distances.

Based on these metrics, the authors developed a text-to-text song lyric generation system using fine-tuned Large Language Models (LLMs). The system extracts features (syllable counts, rhyme scheme, keywords) from source lyrics and uses them to construct prompts for the LLM, which then generates the target lyrics. They explored two generation strategies: generating entire sections at once or generating line by line.

Datasets

  • Parallel Data: A dataset of 69 official English song lyrics and their professional Czech translations for commercial musical films was collected. This dataset comprises 649 parallel song sections.
  • Monolingual Data: A large corpus of 77,478 Czech song sections was scraped from the "Velký zpěvník" (The Great Songbook) webpage, used for training the language models for naturalness and for fine-tuning the LLMs.

Key Insights and Ideas

  • Operationalizing the Pentathlon Principle: The paper successfully translates the qualitative Pentathlon Principle into concrete, computable metrics, enabling objective evaluation of song lyric adaptations.
  • LLMs as Lyric Generators: Fine-tuned LLMs can generate singable lyric adaptations without relying on melody-aligned data or direct bilingual training, demonstrating the power of generative models in this domain.
  • Trade-offs are Inevitable: Human translations show a compromise, particularly in rhyme and sense, suggesting that strict adherence to all Pentathlon aspects is difficult, even for professionals.
  • "Sections" Generation Outperforms "Lines": Generating entire song sections at once yielded better results across most metrics, especially naturalness, compared to generating line by line. This suggests that a broader context is beneficial for coherent and natural lyric generation.
  • Naturalness is King (for Humans): Human evaluators consistently ranked naturalness as the most important aspect of a song lyric adaptation, followed closely by sense and rhythm. Rhyme was deemed the least important.

Key Results

  • The proposed metrics were validated against human-translated lyrics, showing that official translations generally achieve good scores across the Pentathlon Principle, with naturalness being particularly high.
  • The LLM-based generation system, particularly when generating sections, achieved competitive results, outperforming a random baseline and machine translation in singability and rhythm.
  • The "Sections" generation model achieved the best naturalness score among the generated outputs, though it still lagged behind official human translations.
  • Human evaluation showed that participants preferred official translations 79% of the time, followed by generated sections (59%), and machine translations (47%).
  • The study confirmed that human preferences align with the Pentathlon Principle, with a strong emphasis on naturalness and a balanced approach across aspects.

Limitations and Future Work

  • Language Specificity: The current work is limited to the English-Czech language pair. Future work could explore the applicability of the metrics and system to other languages.
  • Training-Inference Mismatch: The system is trained on target language data but infers based on source language input, a necessary compromise due to the lack of bilingual training data.
  • Model Size: The study used smaller LLMs due to resource limitations. Exploring larger models could yield further improvements.
  • Stress Pattern Metric: The authors acknowledge that their rhythm metric (syllable distance) is not perfect and suggest developing a better stress pattern distance metric as future work.
  • Data Release: The code and data are available, but due to copyright, the data is released under a Research Licence.

Summary updated

BibTeX
@inproceedings{stepankova2025song,
  author = {Barbora Štěpánková and Rudolf Rosa},
  title = {{Song Lyrics Adaptations: Computational Interpretation of the Pentathlon Principle}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2025},
}
Figure