Introduction
While deep learning models like Transformers and LSTMs have demonstrated a remarkable ability to generate music, evaluating the quality of these outputs remains a significant challenge. These models often learn style implicitly, leading to an evaluation bottleneck where researchers rely on expensive, subjective, and inconsistent human assessments. To address this, researchers have developed an automated "grading function" designed to objectively evaluate how closely machine-generated music adheres to the specific stylistic constraints of J.S. Bach’s four-part chorales.
Methodology: Statistical Stylistic Evaluation
The proposed grading function, $g(c)$, conceptualizes a musical composition as a collection of statistical distributions across six primary features:
- Pitch and Rhythm: Distributions of scale degrees and note durations.
- Melodic Intervals: Voice-specific analysis of leaps within the Soprano, Alto, Tenor, and Bass parts.
- Harmonic Qualities: The frequency of various chord types (e.g., major, minor, diminished).
- Parallel Errors: Detection of forbidden part-writing moves, such as parallel fifths and octaves.
- Repeated Sequences: The prevalence of recurring motifs.
To quantify stylistic adherence, the system employs the Wasserstein metric (Earth Mover’s Distance) to measure the distance between the feature distribution of a generated chorale and the aggregate distribution of the 351-chorale Bach corpus. A lower aggregate score indicates a higher degree of stylistic fidelity.
As shown in the table above, generated chorales typically exhibit higher mean distances and standard deviations across almost all metrics, particularly in "Parallel Errors" and "Repeated Sequences," where machine models often struggle to replicate Bach's nuanced balance of rules and creativity.
Results and Performance
The grading function was tested against a Transformer model trained on the Bach corpus. In a paired discrimination task—identifying the real Bach piece in a "Bach vs. Machine" pair—the automated function achieved an accuracy of 92.6%.
Remarkably, this performance exceeds that of human musical experts, who achieved 86.7% accuracy. Novice and intermediate listeners performed significantly worse, often failing to distinguish the subtle stylistic markers that the grading function captures. The separation between authentic and generated scores was statistically robust, with a Kolmogorov-Smirnov test p-value of $1e-78$, indicating that the metric is highly effective at distinguishing genuine compositions from machine-generated ones.
Discussion and Future Work
The primary advantage of this grading function is its interpretability. Unlike "black-box" neural classifiers, it provides granular feedback on specific musical dimensions, allowing researchers to pinpoint whether a model fails in harmonic progression, rhythmic variety, or voice-leading.
Future research will focus on optimizing feature weights through human preference learning and extending the framework to other musical genres, such as jazz or pop. Additionally, integrating this grading function as a reward signal in reinforcement learning loops could enable generative models to actively correct stylistic errors during the training process.
Summary updated