Fine-Grained Position Helps Memorizing More, a Novel Music Compound Transformer Model with Feature Interaction Fusion

Zuchao Li, Ruhan Gong, Yineng Chen, Kehua SuAAAI 2023 · DOI

Abstract

Symbolic music understanding often utilizes "Compound Word" (CP) representations to handle multi-attribute musical events. However, standard Transformers suffer from attribute isolation and positional imprecision in this context. We introduce a novel architecture featuring Feature Interaction Fusion (FiF) to capture dynamic correlations between note attributes and Rotational Absolute-Relative (RoAR) position encoding to provide finer-grained structural context. Our model achieves state-of-the-art results across melody, velocity, composer, and emotion classification tasks, demonstrating significant improvements in music representation learning.

Introduction: The Challenges of Symbolic Music

In symbolic music processing (e.g., MIDI), researchers frequently use a "Compound Word" (CP) representation. Instead of treating attributes like pitch, duration, and velocity as separate tokens in a sequence, CP groups them into a single "super-token" per time step to manage sequence length.

Despite its efficiency, the authors identify two critical flaws in existing CP-based Transformers:

  1. Attribute Isolation: Standard models typically concatenate or linearly project attributes, failing to capture the rich internal correlations (e.g., the relationship between a specific pitch and its typical duration).
  2. Positional Imprecision: Music is highly sensitive to ordering. Current position encoding methods, such as Rotary Position Embedding (RoPE), often lose absolute position information during attention calculations, which is detrimental to capturing global musical structure.

Methods: FiF and RoAR

To address these limitations, the proposed architecture introduces two primary innovations:

Feature Interaction Fusion (FiF)

Before entering the main Transformer layers, the FiF module applies a multi-head self-attention mechanism within each compound word. This allows the model to dynamically learn how attributes like "pitch" and "bar-beat" interact for every musical event, creating a more holistic representation than simple concatenation.

Rotational Absolute-Relative (RoAR) Position Encoding

Building on RoPE, the authors argue that standard rotary embeddings effectively convert absolute positions into relative ones, losing global context. RoAR incorporates additional terms into the attention formula that explicitly model absolute-relative interactions. This provides the model with a "finer-grained" sense of an event's position both globally within the piece and locally relative to other notes.

Architecture of the proposed MIDI-FiF-RoAR model, showing the FiF layer and RoAR attention module.

Hybrid Pre-training

The model employs a combination of Masked Language Modeling (MLM) and Causal Language Modeling (CLM). This hybrid approach allows the model to learn robust representations of musical grammar through both bidirectional context and next-note prediction.

Experimental Setup

The study evaluates the model using five diverse datasets:

  • Pop1K7 & ASAP: Large-scale datasets for pre-training.
  • POP909: Benchmarks for melody extraction and velocity prediction.
  • Pianist8: Tasked with identifying one of eight composers.
  • EMOPIA: Categorizing music into four emotional quadrants.

Key Results and Insights

The proposed model, MIDI-FiF-RoAR, consistently outperformed previous state-of-the-art models, including MidiBERT, across all benchmarks.

Performance comparison of MIDI-FiF-RoAR against state-of-the-art models across various music understanding tasks.

The "Small Vocabulary" Problem

A key insight from the research is that music suffers from a "small vocabulary" problem compared to natural language. While NLP vocabularies are vast, music often relies on a limited range (e.g., 128 pitches). This makes position collisions more likely, rendering precise position encoding (like RoAR) far more critical for music than for text.

Performance Gains

  • Emotion Classification: The model showed a dramatic improvement, rising from 69.72% to 76.15%.
  • Composer Classification: Performance increased from 79.37% to 80.95%.
  • Efficiency: Ablation studies confirmed that FiF and RoAR allow the model to maintain high performance even with reduced pre-training data.

Conclusion

By treating musical events as multi-dimensional entities and enhancing positional awareness, the MIDI-FiF-RoAR model significantly advances symbolic music understanding. Future work will focus on structural improvements that further incorporate the "physics" of music—such as rhythm and harmony—directly into the Transformer architecture.

Summary updated

BibTeX
@inproceedings{li2023fine,
  author = {Zuchao Li and Ruhan Gong and Yineng Chen and Kehua Su},
  title = {{Fine-Grained Position Helps Memorizing More, a Novel Music Compound Transformer Model with Feature Interaction Fusion}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2023},
  doi = {10.1609/aaai.v37i4.25650},
}
Figure