ITO-Master: Inference-Time Optimization for Audio Effects Modeling of Music Mastering Processors

Junghyun Koo, Marco A. Martínez Ramírez, Wei-Hsiang Liao, Giorgio Fabbro, Michele Mancusi, Yuki Mitsufuji

TL;DR

ITO-Master introduces an inference-time optimization (ITO) framework for reference-based music mastering style transfer. Instead of relying on a fixed, one-shot transformation, it optimizes the reference embedding ($z_{\text{ref}}$) at inference time using gradient descent. This approach grants fine-grained user control over mastering characteristics across both black-box neural networks and fully differentiable white-box processor chains.

What Problem Does This Paper Address?

Music mastering—applying equalization, compression, stereo imaging, and limiting—ensures commercial readiness and sonic consistency across playback systems. Automated mastering tools have emerged, but reference-based style transfer systems typically perform fixed, unconditioned, or "one-shot" processing based on a reference track.

If the output doesn't quite match an engineer's artistic vision, the user has no direct mechanism to guide or refine the outcome. Existing white-box models (which offer transparency) often rely on oversimplified audio effect chains that struggle with heavily compressed commercial audio, while black-box models lack interpretability and user controllability.

Methods

ITO-Master addresses these gaps by decoupling macro-level style transfer from micro-level inference adjustments:

  1. Dual Converter Architectures ($\Psi$):
  2. Black-box: A Temporal Convolutional Network (TCN) directly predicting output waveforms via multi-scale spectral loss ($L_{\text{MSS}}$).
  3. White-box: A network predicting 46 parameters ($\Theta$) for a realistic, 6-module differentiable mastering chain (6-band parametric EQ, distortion, 3-band compressor with 4th-order Linkwitz-Riley crossovers and differentiable all-pole filters, makeup gain, stereo imager, and limiter). The differentiable all-pole filters allow the model to execute both compression/limiting and expansion/delimiting.
  4. Joint Reference Encoder Training: The reference encoder $\Phi$, initialized from a pre-trained FXencoder, is trained end-to-end alongside $\Psi$ using synthetic pairings generated from unmastered tracks with random FX manipulations and FX-Normalization ($f_{\text{norm}}$).
  5. Inference-Time Optimization (ITO): During inference, rather than fine-tuning model weights or optimizing DSP parameters $\Theta$ from scratch, ITO optimizes the 2048-dimensional reference embedding $z_{\text{ref}}$ via gradient descent: $$z_{\text{ref}}^{(t+1)} = z_{\text{ref}}^{(t)} - \eta \nabla_{z} \mathcal{L}_{\text{aux}}(\Psi(x_{\text{in}}, z_{\text{ref}}^{(t)}), x_{\text{ref}})$$ The auxiliary loss $\mathcal{L}_{\text{aux}}$ can be a content-independent Audio Feature (AF) loss or a multimodal objective like CLAP text-audio embedding similarity.
       +--------------------+
x_ref  | Reference Encoder  |-----> z_ref (Optimized via ITO)
       +--------------------+          |
                                       v
x_in  ------------------------> [ Mastering Converter Ψ ] ----> Mastered Audio y'
                                 (Black-box or White-box)

Datasets

  • Training: MoisesDB unmastered multitrack mixtures, processed with random audio effects pipelines.
  • Validation: MUSDB18-HQ validation subset.
  • Evaluation: 200 tracks randomly sampled from the MTG-Jamendo dataset (100 target tracks $x_{\text{in}}$, 100 reference tracks $x_{\text{ref}}$).
  • No new datasets or benchmarks were introduced.

Key Insights and Ideas

  • Optimizing Latents Beats Optimizing DSP Parameters Directly: Running ITO on $z_{\text{ref}}$ requires $\le 100$ gradient steps and converges stably. Conversely, optimizing the white-box DSP chain parameters $\Theta$ from scratch takes thousands of steps and degrades quality (AF loss 0.250 vs. 0.139). Content-independent loss functions lack the structural context that $\Psi$ learned during training, making direct parameter search prone to bad local minima.
  • The Black-box vs. White-box Trade-off: Black-box models excel at matching granular audio feature metrics (lower AF loss: 0.099 vs. 0.139), but white-box models generate outputs that adhere much closer to real-world audio distributions, as shown by lower Fréchet Audio Distance scores across CLAP, DAC, and EnCodec.
  • Text-Guided Mastering Steering: Because ITO operates on $z_{\text{ref}}$ with arbitrary auxiliary objectives, users can steer mastering stylings using natural language prompts (e.g., maximizing cosine similarity with CLAP text embeddings for "Metal Music", "Classic Music", or "Hip-Hop Music"), producing distinct, musically consistent spectral and dynamic shifts.

Key Results

  • Objective Metrics (Jamendo evaluation):
  • Dynamic Range Control: Proposed models achieved Dynamic Range Variability (DRV) between 0.474 and 0.567 with ITO, significantly outperforming heuristic feature-matching baselines like Matchering (0.823) and Fx-Normalization (0.801), demonstrating the value of differentiable multiband compression and limiting.
  • Distributional Realism: White-box with trained $\Phi$ reached superior Fréchet Audio Distance scores ($\text{FAD}_{\text{CLAP}} = 93.2$, $\text{FAD}_{\text{DAC}} = 101.4$, $\text{FAD}_{\text{EnCodec}} = 38.90$), outperforming prior end-to-end remastering models ($\text{FAD}_{\text{DAC}} = 176.7$).
  • Subjective Listening Tests: In MUSHRA tests with 10 audio production professionals, ITO-enhanced models scored significantly higher in mastering style similarity than baseline models ($p < 0.05$).

Limitations and Future Work

  • FAD vs. Feature-Matching Trade-off: Increasing ITO optimization steps continuously minimizes audio feature distance but can slightly degrade perceptual naturalness (higher FAD), requiring careful stopping criteria.
  • Distortion and Complex Non-linearities: While EQ, stereo width, and loudness are normalized during data preparation, distortion and heavy compression are not easily normalized on-the-fly during training.
  • Garbage-In, Garbage-Out References: Mastering is curatorial; matching an aesthetically unsuitable reference can lead to high similarity metrics but poor musical results. The authors propose integrating perceptual preference metrics and user usability studies in future work.

Summary updated

Abstract

Music mastering style transfer aims to model and apply the mastering characteristics of a reference track to a target track, simulating the professional mastering process. However, existing methods apply fixed processing based on a reference track, limiting users' ability to fine-tune the results to match their artistic intent.In this paper, we introduce the ITO-Master framework, a reference-based mastering style transfer system that integrates Inference-Time Optimization (ITO) to enable finer user control over the mastering process. By optimizing the reference embedding during inference, our approach allows users to refine the output dynamically, making micro-level adjustments to achieve more precise mastering results. We explore both black-box and white-box methods for modeling mastering processors and demonstrate that ITO improves mastering performance across different styles. Through objective evaluation, subjective listening tests, and qualitative analysis using text-based conditioning with CLAP embeddings, we validate that ITO enhances mastering style similarity while offering increased adaptability. Our framework provides an effective and user-controllable solution for mastering style transfer, allowing users to refine their results beyond the initial style transfer.
BibTeX
@inproceedings{koo2025ito,
  author = {Junghyun Koo and Marco A. Martínez Ramírez and Wei-Hsiang Liao and Giorgio Fabbro and Michele Mancusi and Yuki Mitsufuji},
  title = {{ITO-Master: Inference-Time Optimization for Audio Effects Modeling of Music Mastering Processors}},
  booktitle = {ISMIR},
  year = {2025},
  url = {https://openreview.net/forum?id=EerfoTQO5Y},
}

Loading related content...

Figure