Introduction
Re-orchestration is the sophisticated art of adapting a musical composition for a different set of instruments. Unlike simple track assignment, true re-orchestration requires adjusting the musical texture—including rhythmic density, polyphony, and register—while ensuring the primary melody remains recognizable. METEOR introduces a Transformer-based Variational Auto-Encoder (VAE) designed to handle these complexities by combining fine-grained textural controls with a novel melody-enforcement technique.
Problem Statement
Existing AI models for music style transfer often struggle with two primary challenges:
- Melodic Fidelity: Models frequently "hallucinate" or drift away from the original melody during generation.
- Granular Control: Most systems offer only global style controls or are restricted to fixed ensembles, lacking the precision to specify attributes for individual instruments within a large orchestra.
Methodology
METEOR builds upon the MuseMorphose architecture but introduces several key innovations to facilitate multi-track orchestral re-orchestration.
Vertical Tokenization
Instead of processing tracks sequentially, METEOR uses an enhanced REMI+ format with "vertical parsing." By grouping notes by their time position, the model better captures the harmonic relationships and interplay between different instruments in an ensemble.
Hierarchical Control Headers
To provide precise user intervention, each bar of music is preceded by a control header. These headers allow for the specification of both bar-level and track-level attributes.
- Bar-level: Controls overall rhythmic intensity and polyphonicity.
- Track-level: Allows users to set the Average Pitch (register) and Pitch Diversity for every instrument in the ensemble.
Melody-Enforcement Guidance
To ensure the melody is never lost, METEOR employs a "skyline algorithm" to identify the melody in the source piece. During inference, the model is forced to include these melody tokens at the correct time steps, while the autoregressive decoder fills in the accompaniment around them. This allows for "Zero-Shot Lead Sheet Orchestration," where a simple melody and chord progression can be expanded into a complex orchestral movement.
Evaluation and Results
The model was trained on SymphonyNet, a dataset of approximately 46,000 multi-track symbolic music pieces.
Key Metrics
- Melodic Fidelity: METEOR achieved a score of 0.632, significantly outperforming baselines like FIGARO (0.271) and AccoMontage-band (0.338).
- Controllability: The model demonstrated a Spearman correlation of > 0.9 between user-requested attributes (like pitch diversity) and the generated output.
- Instrument Realism: The model successfully respected instrument physical ranges, with 0% out-of-range notes for flute and only ~1.6% for violin.
Conclusion and Future Work
METEOR demonstrates that treating melody as a hard constraint rather than a latent feature is critical for high-fidelity re-orchestration. While the model excels at local texture and instrument realism, future work will address piece-level structural consistency and more complex melody detection beyond the skyline heuristic.
Summary updated