A Comparative Study of LLM Prompting and Fine-Tuning for Cross-genre Authorship Attribution on Chinese Lyrics

Yuxin Li, Lorraine Xu, Meng Fan WangarXiv 2025 · DOI · arXiv

Abstract

Authorship Attribution (AA) in song lyrics presents unique challenges due to stylistic shifts across musical genres. This paper evaluates two technical approaches—zero-shot Large Language Model (LLM) prompting and fine-tuning with contrastive learning—on a newly curated dataset of Chinese lyrics. We find that while LLMs excel in zero-shot settings due to their broad knowledge base, fine-tuning provides superior robustness for cross-genre attribution. Performance varies significantly by genre, with structured themes like Folklore being easier to identify than abstract themes like Romance.

Introduction

Identifying the author of a text, or Authorship Attribution (AA), is a cornerstone of literary analysis and forensics. However, its application to song lyrics, particularly in Chinese, remains under-explored. Lyrics are often short and highly stylized, where the "voice" of the lyricist is frequently masked by the thematic requirements of a specific musical genre. This study investigates whether models can distinguish between "thematic noise" (the song's topic) and the "stylistic signal" (the author's unique signature) across diverse genres.

Methodology

The research team compared two primary strategies:

  1. Zero-Shot LLM Prompting: Utilizing DeepSeek-Llama-70B, the researchers designed linguistically informed prompts. These prompts directed the model to focus on structural features—such as verb usage, punctuation patterns, and rare vocabulary—while explicitly ignoring the thematic content.
  2. Fine-Tuning with Contrastive Learning: A Chinese-RoBERTa base model was fine-tuned using a contrastive loss function. This method trains the model to map lyrics by the same author closer together in vector space, regardless of genre, while pushing lyrics by different authors further apart.

To prepare the data, a rigorous pipeline was established to remove non-stylistic artifacts.

The data preprocessing pipeline used to unify formats and remove outliers.

The pipeline included removing timestamps, special characters, speaker labels, and whitespaces. Additionally, Tukey’s Fences were applied to eliminate length outliers, ensuring models could not rely on song length as a proxy for authorship.

Dataset and Genre Classification

The study introduced a manually curated dataset of 1,055 Chinese songs released post-2000. The data was categorized into five thematic genres: Folklore & Tradition, Love & Romance, Life & Reflection, Society & Reality, and Landscape & Journey.

Distribution of lyrics across genres for the Training, Test1, and Test2 datasets.

The dataset was split into:

  • Training/Test1 Set: 1,000 songs for model development and initial evaluation.
  • Test2 Set: A "hard mode" evaluation set of 55 songs featuring authors not present in the training data.

Key Results and Discussion

The experiments yielded several critical insights:

The Genre Gap

Attribution accuracy is highly dependent on the genre. Folklore & Tradition lyrics, which often feature rigid structures and specific cultural motifs, reached near-perfect attribution scores. In contrast, Love & Romance proved the most challenging, as it often employs generic, sentimental language that blurs individual stylistic differences.

Prompting vs. Fine-Tuning

In Test1 (real-world data), fine-tuning significantly improved F1 Macro scores for cross-genre attribution. However, in Test2 (unseen authors), the zero-shot LLM often maintained a lead in raw accuracy, likely due to the massive pre-existing knowledge base of the 70B parameter model.

Robustness and Stability

The fine-tuned model demonstrated better stability when an author transitioned between genres (e.g., from a "Society" theme to a "Landscape" theme). This suggests that contrastive learning effectively helps the model ignore "genre shortcuts"—the tendency to assume two songs are by the same author simply because they share a topic.

Limitations and Future Directions

The primary constraint of this study is the dataset size (1,055 songs), which is relatively small for deep learning tasks. Furthermore, token-level data augmentation (inserting synthetic tokens) was found to potentially simplify the task in unrealistic ways.

Future research will focus on:

  • Expanding the dataset to include a broader range of authors.
  • Developing more sophisticated data generation techniques beyond token-level augmentation.
  • Exploring domain-adaptive pretraining on a massive lyrics corpus to further refine the model's understanding of lyric-specific stylistic markers.

Summary updated

BibTeX
@misc{li2025comparative,
  author = {Yuxin Li and Lorraine Xu and Meng Fan Wang},
  title = {{A Comparative Study of LLM Prompting and Fine-Tuning for Cross-genre Authorship Attribution on Chinese Lyrics}},
  year = {2025},
  eprint = {2511.21930},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2511.21930},
}
Figure