When we design optimization algorithms, we often think in terms of absolute values. We want to minimize a loss function $f(x)$ or maximize a reward $R(x)$, so we compute gradients based on the exact numerical outputs these functions provide.
However, this reliance on absolute magnitudes introduces a fundamental fragility. Imagine you are optimizing a machine learning model, and someone applies a strictly increasing transformation $g$ to your loss function—such as squaring it, exponentiating it, or applying a non-linear scaling. The underlying geometry of the problem remains identical: the location of the minimum does not change, and the relative ordering of every single point in the search space is perfectly preserved.
Yet, for standard gradient-based optimization, this transformation changes everything. Because $\nabla g(f(x)) = g'(f(x)) \nabla f(x)$, the magnitudes of your gradients are completely warped. An algorithm that converged smoothly on $f(x)$ might explode or stagnate on $g(f(x))$, requiring tedious re-tuning of learning rates and hyperparameters. As noted by Kushagra Gupta et al. in Monotonic Transformation Invariant Multi-task Learning, this sensitivity makes optimization fragile to arbitrary choices in loss function design or reward scaling.
The solution to this problem is monotonic transformation invariance: designing optimization frameworks that discard absolute scalar values entirely, relying instead on the relative ranking or pairwise comparisons of candidate solutions.
The Core Intuition: Level Sets and Ordinal Invariance
Why does a comparison-based approach work? Consider the concept of level sets—the contours of a function where the output value remains constant. If you change the numerical values assigned to these contours but keep their spatial sequence intact, you have performed a monotonic transformation.
In Analysis of a natural gradient algorithm on monotonic convex-quadratic-composite functions, Youhei Akimoto demonstrates that by defining objective functions based on the volume of these level sets rather than raw values, the optimization process becomes completely invariant to strictly increasing transformations. Because the algorithm's trajectory depends solely on the ranking of candidate solutions, its behavior remains identical whether the function outputs raw values or a highly warped, non-linear variation of those values.
By shifting our perspective from value optimization to rank optimization, we gain immense robustness against non-smoothness, poorly calibrated scales, and outliers.
The Mechanism: Fitness Shaping and Quantile Transformations
To achieve this invariance in practice, algorithms use a technique known as fitness shaping or quantile-based transformation. Instead of feeding raw objective scores directly into an optimizer, the population of candidate solutions is sorted, and each solution is assigned a utility value based purely on its rank.
Raw Scores: [ 0.1, 10.4, 10.5, 999.0 ] (Highly skewed by outlier)
Sorted Ranks: [ 1, 2, 3, 4 ]
Rank Utilities: [ -0.75, -0.25, 0.25, 0.75] (Normalized and stable)
In the domain of black-box optimization, Daan Wierstra et al. leverage this in Natural Evolution Strategies. By mapping raw objective values to rank-based utilities, they ensure that the search behavior is driven entirely by the relative ordering of candidates. This theme is expanded by Yi Sun et al. in Efficient Natural Evolution Strategies, where combining natural gradient descent with rank-based transformations allows the same algorithm to handle diverse fitness landscapes without manual tuning for specific problem geometries.
From an information-geometric perspective, Yann Ollivier et al. prove in Objective improvement in information-geometric optimization that transforming objective values into weighted preferences based on quantiles guarantees "monotone improvement." The algorithm consistently moves toward better regions of the search space regardless of how the objective function is stretched or scaled.
Robustness in Black-Box and Global Search
When gradients are unavailable, comparison-based invariance becomes a superpower. In Gradientless Descent: High-Dimensional Zeroth-Order Optimization, Daniel Golovin et al. demonstrate that zeroth-order algorithms relying exclusively on ordinal comparisons are inherently immune to non-linear warping of the output space.
This principle is equally vital when using surrogate models to accelerate optimization. If an optimizer is comparison-based, its surrogate model must be too. Ilya Loshchilov et al. establish in Comparison-Based Optimizers Need Comparison-Based Surrogates that using rank-based surrogates (like rank-based Support Vector Machines) preserves the algorithm's core invariance, preventing arbitrary scaling from destabilizing the approximation.
Even when training neural networks to act as optimizers themselves, sorting the population is key. In B2Opt: Learning to Optimize Black-box Optimization with Little Budget, Kai Wu shows that sorting population members by fitness before passing them to a neural optimizer makes the learned strategy agnostic to absolute scales, allowing it to generalize across completely unseen problem landscapes.
Preventing Domination in Multi-Task and Multi-Objective Systems
The utility of monotonic transformation invariance becomes even more pronounced when dealing with multiple competing objectives. In traditional multi-task learning, balancing different loss functions requires extensive hyperparameter tuning because a single task with an artificially inflated loss scale can easily dominate the shared model updates.
To resolve this, the authors of Monotonic Transformation Invariant Multi-task Learning design an optimization framework that relies on normalized gradients, making the training trajectory mathematically invariant to arbitrary monotonic nonaffine scalings of individual task losses.
A parallel challenge exists in multi-objective Bayesian optimization, where indicators like hypervolume are highly sensitive to how objectives are scaled. In BOtied: Multi-objective Bayesian optimization with tied multivariate ranks, Stephen Ra addresses this by applying the Probability Integral Transform to objective values. This maps raw data scales to multivariate ranks derived from joint cumulative distribution functions, creating a scale-invariant acquisition function that remains robust even when objectives are measured in completely different physical units.
Mitigating Over-Optimization in Preference Learning
Beyond traditional optimization, rank-based invariance has emerged as a crucial mechanism for aligning generative foundation models, such as Large Language Models (LLMs), with human preferences.
Standard Reinforcement Learning from Human Feedback (RLHF) typically trains a scalar reward model to score outputs. However, optimizing directly against a scalar metric often leads to "reward over-optimization"—a phenomenon where the model learns to exploit uncalibrated extreme values or noise in the reward model, causing actual output quality to degrade.
Scalar Reward Optimization: Policy exploits uncalibrated peaks
(High vulnerability to reward model noise)
Rank-Based Optimization: Policy optimizes relative ordering
(Bounded, stable updates based on preference)
To stabilize this process, Hanze Dong et al. introduce RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignment. Instead of utilizing absolute reward magnitudes, RAFT filters and fine-tunes models based on the relative ranking of samples within a "best-of-K" pool, protecting the alignment process from reward scaling anomalies.
Expanding on this paradigm, Yuxuan Tang et al. propose Beyond Pairwise: Empowering LLM Alignment With (Ranked) Choice Modeling. They demonstrate that by focusing on ordinal distance rather than absolute scalar rewards, preference models reduce the risk of over-optimization and provide a significantly more robust training signal.
Conclusion
Whether navigating a non-differentiable black-box landscape, balancing a complex multi-task network, or aligning a foundation model with human values, the core lesson remains the same: absolute values are often an illusion of the chosen representation. By anchoring our optimization algorithms to the invariant reality of rankings and comparisons, we build systems that are inherently stable, robust to noise, and free from the shackles of manual scale tuning.
Key Foundational Work
Algorithms (X, sigma, eta): Quasi-random Mutations for Evolution Strategies (2005) — contributed 2 concept elements to this direction.