ITO-Master: Inference-Time Optimization for Audio Effects Modeling of Music Mastering Processors

Junghyun Koo, Marco A. Martínez Ramírez, Wei-Hsiang Liao, Giorgio Fabbro, Michele Mancusi, Yuki Mitsufuji

Abstract

Music mastering style transfer aims to model and apply the mastering characteristics of a reference track to a target track, simulating the professional mastering process. However, existing methods apply fixed processing based on a reference track, limiting users' ability to fine-tune the results to match their artistic intent. In this paper, we introduce the ITO-Master framework, a reference-based mastering style transfer system that integrates Inference-Time Optimization (ITO) to enable finer user control over the mastering process. By optimizing the reference embedding during inference, our approach allows users to refine the output dynamically, making micro-level adjustments to achieve more precise mastering results. We explore both black-box and white-box methods for modeling mastering processors and demonstrate that ITO improves mastering performance across different styles. Through objective evaluation, subjective listening tests, and qualitative analysis using text-based conditioning with CLAP embeddings, we validate that ITO enhances mastering style similarity while offering increased adaptability. Our framework provides an effective and user-controllable solution for mastering style transfer, allowing users to refine their results beyond the initial style transfer.

Introduction

Music mastering is the critical final stage of audio production, involving a sophisticated chain of equalization, compression, and limiting to ensure a track sounds professional across all playback systems. While automated AI mastering tools have emerged, they typically function as "black boxes," offering a one-size-fits-all output based on a reference track. This lack of transparency and controllability often leaves producers unable to "nudge" the AI toward a specific artistic vision without retraining the entire model.

ITO-Master addresses these limitations by introducing a flexible framework for music mastering style transfer. By utilizing Inference-Time Optimization (ITO), the system allows for dynamic refinement of mastering results after the initial pass. This enables users to achieve professional-grade audio with unprecedented control, whether they are matching a specific reference track or steering the sound via text-based prompts.

The ITO-Master Framework

The core of the system is the Mastering Style Converter ($\Psi$), which processes a target track using a style embedding ($z_{ref}$). The researchers explored two distinct architectural approaches to modeling the mastering chain:

  1. Black-Box Modeling: A Temporal Convolutional Network (TCN) that predicts the mastered waveform directly. This approach excels at capturing high-level technical patterns and minimizing audio feature loss.
  2. White-Box Modeling: A TCN that predicts parameters for a fully differentiable mastering chain. This chain includes a 6-band EQ, distortion, 3-band compressor, stereo imager, and limiter. This approach is highly interpretable, as it adjusts actual "knobs" on a virtual mastering rack.

Inference-Time Optimization (ITO)

The "secret sauce" of this framework is the optimization process that occurs during inference. Instead of simply accepting the first output from the model, ITO-Master treats the mastering process as an iterative optimization problem. The system updates the style embedding ($z_{ref}$) to minimize a specific loss function, such as Audio Feature loss or CLAP text-audio similarity.

A key finding was that optimizing the style embedding is significantly more effective than attempting to optimize the processor parameters from scratch. The embedding provides a "warm start," allowing the model to reach a refined, high-quality result in relatively few steps (typically 100 or fewer).

Text-Steered Mastering and Semantic Control

One of the most innovative aspects of ITO-Master is its ability to handle text-based conditioning. By integrating CLAP (Contrastive Language-Audio Pretraining) embeddings, which link natural language to audio features, users can bypass the need for a reference audio file entirely. A user can input a prompt like "warm vintage sound" or "punchy hip-hop," and the ITO process will steer the mastering chain to match the semantic description. This bridges the gap between technical audio engineering and creative intent.

Experimental Setup and Datasets

The model was trained and validated using several industry-standard datasets:

  • MoisesDB & MUSDB18: These datasets provided the necessary stems and unmastered mixes. The researchers created synthetic "mastered" versions by applying randomized effects chains to generate training data.
  • MTG-Jamendo: This large-scale dataset was used for final evaluation, testing the model's performance across a diverse range of real-world genres and musical styles.

Results and Discussion

Objective Performance

In the white-box configuration, the addition of ITO led to a significant improvement in mastering consistency. The Dynamic Range Variability (DRV)—a metric used to measure how well the mastering matches the reference's dynamics—dropped from 0.598 to 0.474. This outperformed established baselines such as Matchering and E2E Remastering.

Subjective Quality

MUSHRA-style listening tests conducted with ten audio experts confirmed the objective findings. Participants consistently rated the ITO-Master outputs as more similar to the reference tracks than outputs from baseline automated systems. Furthermore, while black-box models often achieved lower technical loss, white-box models were frequently preferred for their "natural" and "musical" sound quality, as reflected in lower Fréchet Audio Distance (FAD) scores.

Conclusion and Future Work

ITO-Master represents a significant step forward in making AI mastering a collaborative tool rather than a rigid automation. By allowing for inference-time adjustments, it gives creators the micro-level control required for professional production.

Future research will focus on improving the user interface to make these micro-adjustments more intuitive for non-engineers. Additionally, the team aims to address "reference alignment" issues, ensuring the model can better handle cases where the reference and target tracks have vastly different song structures or arrangements. Despite these challenges, ITO-Master provides a robust foundation for the next generation of user-controllable audio effects modeling.

Summary updated

Abstract

Music mastering style transfer aims to model and apply the mastering characteristics of a reference track to a target track, simulating the professional mastering process. However, existing methods apply fixed processing based on a reference track, limiting users' ability to fine-tune the results to match their artistic intent. In this paper, we introduce the ITO-Master framework, a reference-based mastering style transfer system that integrates Inference-Time Optimization (ITO) to enable finer user control over the mastering process. By optimizing the reference embedding during inference, our approach allows users to refine the output dynamically, making micro-level adjustments to achieve more precise mastering results. We explore both black-box and white-box methods for modeling mastering processors and demonstrate that ITO improves mastering performance across different styles. Through objective evaluation, subjective listening tests, and qualitative analysis using text-based conditioning with CLAP embeddings, we validate that ITO enhances mastering style similarity while offering increased adaptability. Our framework provides an effective and user-controllable solution for mastering style transfer, allowing users to refine their results beyond the initial style transfer.
BibTeX
@inproceedings{koo2025ito,
  author = {Junghyun Koo and Marco A. Martínez Ramírez and Wei-Hsiang Liao and Giorgio Fabbro and Michele Mancusi and Yuki Mitsufuji},
  title = {{ITO-Master: Inference-Time Optimization for Audio Effects Modeling of Music Mastering Processors}},
  booktitle = {ISMIR},
  year = {2025},
  url = {https://openreview.net/forum?id=EerfoTQO5Y},
}
Figure