METEOR: Melody-aware Texture-controllable Symbolic Music Re-Orchestration via Transformer VAE

Dinh-Viet-Toan Le, Yi-Hsuan YangIJCAI 2025 · DOI

Introduction

Re-orchestration is the sophisticated art of adapting a musical composition for a different set of instruments. Unlike simple track assignment, true re-orchestration requires adjusting the musical texture—including rhythmic density, polyphony, and register—while ensuring the primary melody remains recognizable. METEOR introduces a Transformer-based Variational Auto-Encoder (VAE) designed to handle these complexities by combining fine-grained textural controls with a novel melody-enforcement technique.

Figure 1: Comparison of re-orchestration results across different target ensembles and intensities while preserving the reference melody (highlighted in red).

Problem Statement

Existing AI models for music style transfer often struggle with two primary challenges:

  1. Melodic Fidelity: Models frequently "hallucinate" or drift away from the original melody during generation.
  2. Granular Control: Most systems offer only global style controls or are restricted to fixed ensembles, lacking the precision to specify attributes for individual instruments within a large orchestra.

Methodology

METEOR builds upon the MuseMorphose architecture but introduces several key innovations to facilitate multi-track orchestral re-orchestration.

Vertical Tokenization

Instead of processing tracks sequentially, METEOR uses an enhanced REMI+ format with "vertical parsing." By grouping notes by their time position, the model better captures the harmonic relationships and interplay between different instruments in an ensemble.

Hierarchical Control Headers

To provide precise user intervention, each bar of music is preceded by a control header. These headers allow for the specification of both bar-level and track-level attributes.

  • Bar-level: Controls overall rhythmic intensity and polyphonicity.
  • Track-level: Allows users to set the Average Pitch (register) and Pitch Diversity for every instrument in the ensemble.
Figure 6: The hierarchical control header structure allows for specific attribute injection at both the bar and individual track levels.

Melody-Enforcement Guidance

To ensure the melody is never lost, METEOR employs a "skyline algorithm" to identify the melody in the source piece. During inference, the model is forced to include these melody tokens at the correct time steps, while the autoregressive decoder fills in the accompaniment around them. This allows for "Zero-Shot Lead Sheet Orchestration," where a simple melody and chord progression can be expanded into a complex orchestral movement.

Evaluation and Results

The model was trained on SymphonyNet, a dataset of approximately 46,000 multi-track symbolic music pieces.

Key Metrics

  • Melodic Fidelity: METEOR achieved a score of 0.632, significantly outperforming baselines like FIGARO (0.271) and AccoMontage-band (0.338).
  • Controllability: The model demonstrated a Spearman correlation of > 0.9 between user-requested attributes (like pitch diversity) and the generated output.
  • Instrument Realism: The model successfully respected instrument physical ranges, with 0% out-of-range notes for flute and only ~1.6% for violin.

Conclusion and Future Work

METEOR demonstrates that treating melody as a hard constraint rather than a latent feature is critical for high-fidelity re-orchestration. While the model excels at local texture and instrument realism, future work will address piece-level structural consistency and more complex melody detection beyond the skyline heuristic.

Summary updated

BibTeX
@inproceedings{le2025meteor,
  author = {Dinh-Viet-Toan Le and Yi-Hsuan Yang},
  title = {{METEOR: Melody-aware Texture-controllable Symbolic Music Re-Orchestration via Transformer VAE}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.24963/ijcai.2025/1125},
}
Figure