TL;DR
This paper introduces Tradformer, a Transformer-based model for generating traditional Irish and Swedish dance music transcriptions. By carefully engineering sampling strategies and analyzing model internals, Tradformer achieves comparable or better performance than previous LSTM models with significantly fewer parameters, demonstrating the effectiveness of Transformers for this domain.
What Problem Does This Paper Address?
Generating realistic and stylistically coherent music is a long-standing challenge in AI. Traditional music, particularly dance music, often features repetitive structures and long-range dependencies, making it a good candidate for sequence modeling. Previous state-of-the-art models for this task, like the LSTM-based folk-rnn, were successful but computationally intensive. The authors sought to explore whether the Transformer architecture, known for its ability to capture long-range dependencies and its parallelizability, could achieve similar or better results with fewer parameters, specifically for Irish and Swedish traditional dance music. They also aimed to understand the engineering challenges and insights involved in applying Transformers to this domain.
Methods
The core of Tradformer is the Transformer architecture, specifically leveraging its self-attention mechanism. Unlike recurrent models that process sequences step-by-step, Transformers can attend to all parts of the input sequence simultaneously, allowing them to better capture long-range musical structures. The authors detail several key engineering decisions:
- Vocabulary and Representation: They use a similar token-based representation as previous LSTM models, encoding meter, mode, pitch, and duration.
- Model Architecture: Tradformer uses 16 Transformer layers with 4 attention heads each. The embedding dimension was set to 128, found to be a good balance between performance and complexity.
- Sampling Strategy: A crucial aspect was moving beyond naive sampling. They adopted a combination of beam search and nucleus sampling, which significantly improved the quality and coherence of generated tunes by focusing on more probable token sequences.
- Internal Analysis: The authors emphasize the importance of visualizing attention heads and model parameters during training to understand what the model is learning and to guide engineering choices.
Datasets
- Irish Traditional Dance Music: The primary dataset consists of 23,636 transcriptions of traditional Irish dance music, sourced from the folk-rnn dataset.
- Swedish Traditional Dance Music (Slängpolska): For transfer learning, a dataset of 593 slängpolska tunes was scraped from folkwiki.se. This dataset was smaller and presented unique challenges due to its specific musical characteristics.
Key Insights and Ideas
- Transformer Suitability: The Transformer architecture is well-suited for modeling the repetitive and long-range structures found in traditional dance music, potentially outperforming LSTMs with fewer parameters.
- Sampling is Crucial: The choice of sampling strategy has a profound impact on the musical quality of generated output, with beam search and nucleus sampling proving far superior to naive methods.
- Internal Visualization is Key: Visualizing attention patterns and parameter weights provides invaluable insights into the model's learning process, helping to diagnose issues and guide improvements.
- Human Evaluation is Essential: Relying solely on metrics like cross-entropy is insufficient. Human evaluation by musicians is critical for assessing musical quality and making informed design decisions.
- Transfer Learning for Style Adaptation: Tradformer can be effectively fine-tuned on smaller, style-specific datasets, demonstrating its adaptability.
Key Results
- Parameter Efficiency: Tradformer has approximately one-fifth the parameters of the LSTM folk-rnn model.
- Performance: Tradformer achieved comparable or better quality than folk-rnn for Irish traditional music.
- Competition Success: In the AI Music Generation Challenge 2021, Tradformer's generated slängpolska tunes were rated the most favorable by human judges.
- Sampling Impact: The shift to beam search and nucleus sampling led to a significant improvement in generated tune quality, reducing errors and improving coherence.
- Rejection Sampling: For the Swedish challenge, rejection sampling was used to filter out low-quality outputs, discarding about one-tenth of generated tunes.
Limitations and Future Work
- Maximum Sequence Length: A limitation of the Transformer architecture is its fixed maximum sequence length, which is not ideal for potentially very long musical pieces. The authors suggest this as a direction for future work.
- Balancing Correctness and Variety: The authors note the inherent difficulty in simultaneously achieving musical correctness and "surprising" or varied outputs.
- Data Scarcity: For the Swedish music task, the limited dataset size led to issues like excessive repetition and occasional "stuck" cycles.
- Further Engineering: The paper suggests that further exploration of sampling strategies and model architectures could yield even better results.
Summary updated