Incorporating Music Knowledge in Continual Dataset Augmentation for Music Generation

Alisa Liu, Alexander Fang, Gaëtan Hadjeres, Prem Seetharaman, Bryan PardoarXiv 2020 · arXiv

Introduction

In the domain of creative AI, data scarcity presents a significant hurdle. For specialized tasks like composing in the style of J.S. Bach, researchers are restricted to a finite historical corpus—specifically, the 351 extant chorales. This limited data often leads to rapid overfitting, where models memorize specific patterns rather than internalizing the underlying musical grammar. To address this, we introduce AUG-GEN (Augmentative Generation), a framework that enables models to "self-augment" their training data using an external, music-theory-based critic.

The AUG-GEN Framework

AUG-GEN employs a "generate-grade-train" loop to iteratively expand the training set with high-quality synthetic examples. The process consists of four primary stages:

  1. Generation: A Transformer model with relative attention generates new musical sequences.
  2. Grading: An external grading function evaluates these sequences based on 18th-century counterpoint rules. This non-differentiable "critic" acts as an objective gatekeeper.
  3. Filtering: Only unique generations that meet a specific quality threshold ($t$) are added to the training set.
  4. Training: The model is retrained on the augmented dataset, effectively learning from its own validated successes.

Comparative Performance

The effectiveness of AUG-GEN was evaluated against several baselines: a model with no augmentation (Baseline-none) and a model that added all generations regardless of quality (Baseline-all). Results indicate that AUG-GEN significantly outperforms both baselines in terms of output quality.

Grade Distribution of Generations from Different Models

As shown in Figure 1, AUG-GEN achieved a median grade of 8.02, substantially closer to the original Bach chorales (median 4.91) than the Baseline-none (10.62) or the noisy Baseline-all (24.10). This confirms that the grading function serves as a vital "low-pass filter," preventing the degradation of model performance by excluding low-quality synthetic data.

Training Dynamics

A key advantage of AUG-GEN is its ability to delay overfitting. While the baseline model reached its lowest validation loss at epoch 16, the AUG-GEN model continued to improve until epoch 19. By this point, synthetic examples constituted approximately 12% of the training data.

Grade Distribution of Generations During Aug-Gen Training

The training progression (Figure 2) illustrates a clear downward trend in error scores over time. The model progressively learns to generate sequences that satisfy the music-theory constraints, with the median grade eventually approaching the third quartile ($Q_3$) of the original Bach dataset.

Conclusion and Future Work

AUG-GEN demonstrates that incorporating domain-specific expert knowledge into the training loop can mitigate the limitations of small datasets. By using an external critic to curate self-generated data, models can achieve better generalization and higher output quality. Future research will focus on expanding the grading function to include global musical structures, such as rhythmic diversity and harmonic modulation, and exploring the framework's applicability to other rule-based domains like code generation.

Summary updated

Abstract

Deep learning has rapidly become the state-of-the-art approach for music generation. However, training a deep model typically requires a large training set, which is often not available for specific musical styles. In this paper, we present augmentative generation (Aug-Gen), a method of dataset augmentation for any music generation system trained on a resource-constrained domain. The key intuition of this method is that the training data for a generative system can be augmented by examples the system produces during the course of training, provided these examples are of sufficiently high quality and variety. We apply Aug-Gen to Transformer-based chorale generation in the style of J.S. Bach, and show that this allows for longer training and results in better generative output.
BibTeX
@misc{liu2020incorporating,
  author = {Alisa Liu and Alexander Fang and Gaëtan Hadjeres and Prem Seetharaman and Bryan Pardo},
  title = {{Incorporating Music Knowledge in Continual Dataset Augmentation for Music Generation}},
  year = {2020},
  eprint = {2006.13331},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2006.13331},
}
Figure