Bach or Mock? A Grading Function for Chorales in the Style of J.S. Bach

Alexander Fang, Alisa Liu, Prem Seetharaman, Bryan PardoarXiv 2020 · arXiv

Introduction

While deep learning models like Transformers and LSTMs have demonstrated a remarkable ability to generate music, evaluating the quality of these outputs remains a significant challenge. These models often learn style implicitly, leading to an evaluation bottleneck where researchers rely on expensive, subjective, and inconsistent human assessments. To address this, researchers have developed an automated "grading function" designed to objectively evaluate how closely machine-generated music adheres to the specific stylistic constraints of J.S. Bach’s four-part chorales.

Methodology: Statistical Stylistic Evaluation

The proposed grading function, $g(c)$, conceptualizes a musical composition as a collection of statistical distributions across six primary features:

  1. Pitch and Rhythm: Distributions of scale degrees and note durations.
  2. Melodic Intervals: Voice-specific analysis of leaps within the Soprano, Alto, Tenor, and Bass parts.
  3. Harmonic Qualities: The frequency of various chord types (e.g., major, minor, diminished).
  4. Parallel Errors: Detection of forbidden part-writing moves, such as parallel fifths and octaves.
  5. Repeated Sequences: The prevalence of recurring motifs.

To quantify stylistic adherence, the system employs the Wasserstein metric (Earth Mover’s Distance) to measure the distance between the feature distribution of a generated chorale and the aggregate distribution of the 351-chorale Bach corpus. A lower aggregate score indicates a higher degree of stylistic fidelity.

Statistical comparison of musical features between authentic Bach chorales and generated samples.

As shown in the table above, generated chorales typically exhibit higher mean distances and standard deviations across almost all metrics, particularly in "Parallel Errors" and "Repeated Sequences," where machine models often struggle to replicate Bach's nuanced balance of rules and creativity.

Results and Performance

The grading function was tested against a Transformer model trained on the Bach corpus. In a paired discrimination task—identifying the real Bach piece in a "Bach vs. Machine" pair—the automated function achieved an accuracy of 92.6%.

Proportion of chorales identified as Bach by human participants and the automated grading function.

Remarkably, this performance exceeds that of human musical experts, who achieved 86.7% accuracy. Novice and intermediate listeners performed significantly worse, often failing to distinguish the subtle stylistic markers that the grading function captures. The separation between authentic and generated scores was statistically robust, with a Kolmogorov-Smirnov test p-value of $1e-78$, indicating that the metric is highly effective at distinguishing genuine compositions from machine-generated ones.

Discussion and Future Work

The primary advantage of this grading function is its interpretability. Unlike "black-box" neural classifiers, it provides granular feedback on specific musical dimensions, allowing researchers to pinpoint whether a model fails in harmonic progression, rhythmic variety, or voice-leading.

Future research will focus on optimizing feature weights through human preference learning and extending the framework to other musical genres, such as jazz or pop. Additionally, integrating this grading function as a reward signal in reinforcement learning loops could enable generative models to actively correct stylistic errors during the training process.

Summary updated

Abstract

Deep generative systems that learn probabilistic models from a corpus of existing music do not explicitly encode knowledge of a musical style, compared to traditional rule-based systems. Thus, it can be difficult to determine whether deep models generate stylistically correct output without expert evaluation, but this is expensive and time-consuming. Therefore, there is a need for automatic, interpretable, and musically-motivated evaluation measures of generated music. In this paper, we introduce a grading function that evaluates four-part chorales in the style of J.S. Bach along important musical features. We use the grading function to evaluate the output of a Transformer model, and show that the function is both interpretable and outperforms human experts at discriminating Bach chorales from model-generated ones.
BibTeX
@misc{fang2020bach,
  author = {Alexander Fang and Alisa Liu and Prem Seetharaman and Bryan Pardo},
  title = {{Bach or Mock? A Grading Function for Chorales in the Style of J.S. Bach}},
  year = {2020},
  eprint = {2006.13329},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2006.13329},
}
Figure