Unsupervised Lead Sheet Generation via Semantic Compression

Zachary Novack, Nikita Srivatsan, Taylor Berg-Kirkpatrick, Julian J. McAuleyarXiv 2023 · DOI · arXiv

Introduction

In music theory and practice, a lead sheet serves as a condensed blueprint of a musical score, capturing only the essential melody and harmonic structure. While these representations are critical for tasks such as automatic accompaniment and arrangement, high-quality datasets pairing full multitrack scores with human-annotated lead sheets are scarce. Traditional approaches often rely on the "skyline algorithm"—a heuristic that selects the highest note at any given time—which frequently fails in complex polyphonic genres like jazz or classical music.

To address this data sparsity, researchers from UC San Diego introduced Lead-AE, an unsupervised framework that treats lead sheet generation as a task of "semantic compression."

Methodology: The Lead-AE Framework

Lead-AE utilizes an Auto-Encoder architecture to learn which musical elements are most significant without explicit supervision. The system is composed of two primary modules:

  1. Score2Lead (Encoder): This module processes a full multitrack score and assigns an importance score to every note and chord. A key innovation is the use of a differentiable top-$k$ operator, which allows the model to select a sparse subset of notes (e.g., the top 10% per onset) while maintaining gradient flow for end-to-end training.
  2. Lead2Score (Decoder): This module attempts to reconstruct the original multitrack arrangement using only the sparse lead sheet produced by the encoder.
Figure 1: The Lead-AE architecture, featuring the Score2Lead encoder with differentiable top-k masking and the Lead2Score decoder.

By rewarding the model for accurate reconstruction, the encoder is forced to identify the "essential" notes that best describe the overall musical structure. The model is "warm-started" by pre-training it to mimic the skyline algorithm before evolving its own selection strategy.

Key Insights

  • Music as Compression: The framework posits that a lead sheet is a latent representation containing sufficient information to reconstruct a full arrangement.
  • Differentiable Selection: Moving beyond deterministic rules allows the model to adapt to context, potentially identifying important middle-voice lines that heuristics would miss.
  • Local Sparsity Constraints: Enforcing a top-$k$ constraint per onset ensures the generated lead sheet remains human-readable and musically coherent.

Experimental Results

The model was evaluated on the Symbolic Orchestral Database (SOD) for complex compression and the POP909 dataset for melody recovery.

Quantitative Performance

On the dense orchestral scores of the SOD dataset, Lead-AE significantly outperformed the skyline baseline across multiple metrics. Specifically, it achieved a Music Translation Evaluation (MuTE) score of 64.04, compared to the baseline's 59.95.

Figure 3: Quantitative comparison on the SOD dataset showing Lead-AE outperforming the Skyline baseline in MuTE and Jaccard similarity metrics.

Human Evaluation

In a "Reading Study" involving professional musicians, Lead-AE’s outputs were preferred over the skyline baseline by a substantial margin. Experts rated Lead-AE at 80% for accuracy and 56.7% for musical fluency, highlighting its ability to capture musically meaningful content that simple heuristics overlook.

Conclusion and Future Work

Lead-AE demonstrates that the inherent structure of music can provide sufficient supervision for learning musical importance. While the current model relies on off-the-shelf chord extractors and unified instrumentation, future work aims to integrate end-to-end chord recognition and multi-instrument "sketches." This approach paves the way for training advanced generative models on high-quality, unsupervised lead sheets.

Summary updated

Abstract

Lead sheets have become commonplace in generative music research, being used as an initial compressed representation for downstream tasks like multitrack music generation and automatic arrangement. Despite this, researchers have often fallen back on deterministic reduction methods (such as the skyline algorithm) to generate lead sheets when seeking paired lead sheets and full scores, with little attention being paid toward the quality of the lead sheets themselves and how they accurately reflect their orchestrated counterparts. To address these issues, we propose the problem of conditional lead sheet generation (i.e. generating a lead sheet given its full score version), and show that this task can be formulated as an unsupervised music compression task, where the lead sheet represents a compressed latent version of the score. We introduce a novel model, called Lead-AE, that models the lead sheets as a discrete subselection of the original sequence, using a differentiable top-k operator to allow for controllable local sparsity constraints. Across both automatic proxy tasks and direct human evaluations, we find that our method improves upon the established deterministic baseline and produces coherent reductions of large multitrack scores.
BibTeX
@misc{novack2023unsupervised,
  author = {Zachary Novack and Nikita Srivatsan and Taylor Berg-Kirkpatrick and Julian J. McAuley},
  title = {{Unsupervised Lead Sheet Generation via Semantic Compression}},
  year = {2023},
  eprint = {2310.10772},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2310.10772},
}
Figure