Six Dragons Fly Again: Reviving 15th-Century Korean Court Music with Transformers and Novel Encoding

Danbinaerin Han, Mark R. H. Gotham, Dongmin Kim, Hannah Park, Sihun Lee, Dasaem Jeong

TL;DR

This paper tackles the challenge of reviving ancient Korean court music, specifically two 15th-century pieces with rudimentary notation. They developed a novel "Jeonggan-like" encoding scheme and used transformer models to generate full orchestral arrangements, demonstrating that AI can successfully reconstruct and reimagine traditional music even with limited data.

What Problem Does This Paper Address?

Reviving historical music often faces the hurdle of incomplete or archaic notation systems. This paper focuses on two 15th-century Korean court music pieces, Chihwapyeong and Chwipunghyeong, whose surviving scores are essentially just basic melodies lacking detailed instrumentation or rhythmic nuances. The goal was to transform these sparse melodies into full, performable orchestral arrangements for six instruments, preserving the essence of the original while making it accessible for modern performance. This is particularly challenging due to the unique heterophonic structure of Korean court music and the limitations of existing music generation models when dealing with such specialized traditions and scarce data.

Methods

The core of the methodology involves several key components:

  • Novel Encoding Scheme (Jeonggan-like): Recognizing that traditional Western encoding methods (like ABC notation) don't capture the nuances of Korean court music notation (Jeongganbo), the authors devised a "Jeonggan-like" (JG-like) encoding. This scheme directly mirrors Jeongganbo's structure, representing notes by their position within a time unit (Jeonggan) and their pitch. Crucially, it incorporates a "beat counter" embedding, which explicitly encodes temporal position (measure, beat, sub-beat) into the model's input. This is a departure from standard positional embeddings and is shown to be vital for understanding musical context.
  • Transformer Models: Two main transformer architectures were employed:
  • Orchestral Part Generation: An encoder-decoder transformer model was used to generate melodies for different instruments based on an input melody. This model leverages self-attention to capture long-term dependencies and musical context.
  • Melody Transformation (BERT-like MLM): To handle the sparse 15th-century melody, a BERT-like masked language model (MLM) was trained. This model learns to "fill in the blanks" by predicting masked tokens, allowing for the transformation of the rudimentary melody into a more complete form suitable for the orchestral transformer. A piano-roll-like encoding was used for the MLM to manage pre-defined token sequences.
  • Optical Music Recognition (OMR): To create a machine-readable dataset, OMR techniques were applied to digitized manuscripts of Korean court music.

Datasets

  • Jeongganbo Dataset: A custom dataset was constructed by applying OMR to 85 pieces of Korean court music, resulting in 28,010 jeonggans (or 141,820 jeonggans when counting each instrument part). This dataset is noted as the first machine-readable Jeongganbo dataset.
  • 15th-Century Music: The specific pieces Chihwapyeong and Chwipunghyeong were used as the target for revival, with their rudimentary melodies serving as input for the transformation process.

Key Insights and Ideas

  • Encoding is Paramount: The paper strongly emphasizes that the choice of encoding scheme is critical for success in specialized music generation tasks. The proposed JG-like encoding, by directly reflecting the source notation's temporal structure, significantly outperforms more generic encodings like ABC-like for complex traditional music.
  • Beat Counter is Essential: Explicitly encoding temporal position via a "beat counter" embedding proved crucial for the transformer model to properly understand musical context, especially with limited training data. This suggests that for models to grasp music beyond simple sequential order, explicit temporal cues are highly beneficial.
  • BERT for Sparse Data: Using a BERT-like MLM for melody transformation demonstrates its effectiveness in enriching sparse musical data, making it suitable for downstream generation tasks. This is a valuable approach for historical music where original scores might be incomplete.
  • Refinement Improves Quality: A post-generation refinement step, where each instrument's melody is regenerated with the context of other instruments, significantly improved the overall quality, particularly in terms of sigimsae (ornamentation) and harmony.

Key Results

  • Objective Evaluation: The JG-like encoding achieved higher F1-scores for generating daegeum melodies, indicating better performance on complex and varied musical content. While ABC-like encoding performed best for the simpler geomungo melodies, JG-like encoding proved more robust for the more intricate parts.
  • Beat Counter Efficacy: Models incorporating the beat counter embedding showed significantly better length match rates compared to those without, demonstrating its importance for generating musically complete sequences.
  • Subjective Evaluation: Expert reviews from the Court Music Orchestra of the National Gugak Center indicated that the generated music was well-represented in terms of genre-specific rhythm and melodic flow, with good ornamentation and melodic progressions. The refinement step notably improved scores for sigimsae, rhythm, and harmony.
  • Performable Output: The generated music was performed by the National Gugak Center orchestra, validating the practical applicability of the AI-generated arrangements.

Limitations and Future Work

  • Limited Data: The authors acknowledge that the models were trained on limited data, which is a common challenge in historical music revival.
  • Model Architecture: While transformers were used, the specific depth and complexity of the models (e.g., 6 layers) might be further optimized.
  • Future Work: The paper suggests that the developed Jeongganbo dataset and the interactive web demo are valuable resources for future research in computational ethnomusicology and music generation. The web demo allows users to input their own melodies for orchestration, expanding the accessibility of traditional Korean music. Further exploration of different encoding schemes and model architectures for diverse traditional music genres is also implied.

Summary updated

BibTeX
@inproceedings{han2024six,
  author = {Danbinaerin Han and Mark R. H. Gotham and Dongmin Kim and Hannah Park and Sihun Lee and Dasaem Jeong},
  title = {{Six Dragons Fly Again: Reviving 15th-Century Korean Court Music with Transformers and Novel Encoding}},
  booktitle = {Ismir},
  year = {2024},
  doi = {10.5281/zenodo.14877313},
}
Figure