Normalizing Heterogeneous Reward Scales Across Multiple Agents

22 concepts18 papers1 proposal

Imagine an autonomous shuttle trying to balance the preferences of two passengers. Passenger A wants to reach their destination quickly and expresses this preference via an internal reward function that outputs values between 0 and 100. Passenger B prioritizes a smooth, comfortable ride, but their internal reward function only outputs values between 0 and 1.

If the shuttle's AI tries to maximize collective welfare by simply adding these two reward functions together, Passenger A’s preferences will completely overwhelm Passenger B's. This distortion does not happen because Passenger A's needs are inherently more urgent or important, but simply because their internal reporting scale is numerically larger.

This scenario illustrates the challenge of interpersonal utility comparison and reward scale misalignment. When aggregating preferences, negotiating deals, or aligning AI systems with a diverse group of humans, we lack a standardized, objective unit to measure and compare internal subjective states. Without a deliberate mechanism to normalize these scales, collective decision-making algorithms inadvertently succumb to the "tyranny of the loud"—prioritizing whoever reports their utility on the largest numerical scale.

The Mathematical Trap of Naive Aggregation

In single-agent decision theory, an agent's utility function is unique only up to a positive affine transformation ($u'(x) = a \cdot u(x) + b$). Multiplying an individual's reward function by 100 changes nothing about their optimal policy. However, the moment multiple agents interact, these arbitrary scaling choices become catastrophic.

As Parand A. Alamdari et al. demonstrate in Policy Aggregation, aggregating individual reward functions is fundamentally biased by these arbitrary numerical scales. This scale heterogeneity introduces severe vulnerabilities in multi-agent reinforcement learning (MARL). For instance, Manisha Senadeera et al. note in EMOTE: An Explainable Architecture for Modelling the Other through Empathy that inferred rewards for other agents often lack a consistent scale, making it incredibly difficult for a learning agent to accurately weigh a peer's preferences against its own internal values.

Even when we attempt to calibrate agents by establishing shared baseline anchors, the math can still fail. In On the Foundations of Expected Expected Utility, Craig Boutilier proves that even if multiple users share identical extreme preference anchors (i.e., agreeing on the absolute best and worst possible outcomes), the relative strength of their preferences for intermediate outcomes can remain completely incomparable.

This problem stretches from algorithmic design back to the roots of political philosophy. Елена Малахова traces this dilemma back to Jeremy Bentham’s utilitarianism, highlighting that the core obstacle to treating governance as a calculable science is the lack of a standardized metric to compare subjective human experiences. Whether an AI is trying to safely run a government (Frédéric Berdoz et al.), distribute public resources fairly (Benjamin Plaut et al.), or optimize two-sided recommendation rankings (Virginie Do et al.), it runs into the same wall: as Jessica Finocchiaro et al. argue, weighting the utilities of different individuals remains an open ethical and political challenge rather than a purely mathematical one.

To build multi-agent systems that are truly equitable and stable, researchers have developed three core algorithmic mechanisms to resolve scale misalignment.

   Naive Aggregation:      [Agent A: 0 to 100]  +  [Agent B: 0 to 1]   --> Agent A Dominates

   1. Fixed Normalization: [Agent A: Sums to 1] +  [Agent B: Sums to 1] --> Equitable Weighting
   2. Distortion Learning: [Agent A Scale]     <-- (Shared History) --> [Translate to Agent B Scale]
   3. Directional Oracles: [Agent A: Move Up]   and [Agent B: Move Left] --> Bypasses Scales Entirely

Mechanism 1: Bounding and Normalization Guarantees

The most straightforward way to prevent one agent from dominating a collective policy is to force all utility functions into a shared mathematical box. A common approach in computational social choice is to normalize utilities so that every agent's preferences sum to a fixed constant (e.g., 1).

While critics often dismiss this practice as an arbitrary mathematical convenience, the authors of Justifications of welfare guarantees under normalized utilities provide a robust theoretical defense. Haris Aziz 0001 et al. prove that achieving a welfare guarantee under normalized utilities is mathematically equivalent to achieving a robust form of Pareto efficiency—one that holds true regardless of how agents scale their internal utilities. Normalization acts as a shield, ensuring that a policy remains fair even if agents alter their reporting metrics.

In practical AI alignment, such as personalizing Reinforcement Learning from Human Feedback (RLHF) for diverse populations, bounding scales is vital for optimization stability. Abhishek Gupta et al. implement this by utilizing a pairwise preference likelihood to bound and normalize rewards across a diverse population of user latents, preventing users with extreme feedback styles from destabilizing the global learning model.

Mechanism 2: Learning Distortion and Scaling Factors

What if we cannot force agents to normalize their rewards? If agents operate in a shared environment and evaluate the same outcomes, we can dynamically learn the translation rate between their internal scales.

Consider a collaborative learning network where agents observe each other's rewards to accelerate their own learning. In Group Learning and Opinion Diffusion in a Broadcast Network, Liu Yang et al. demonstrate that heterogeneous preferences can be reconciled in real-time by estimating a scaling factor between users based on overlapping historical observations. By calculating a "distortion factor"—the exact ratio between a user's local reward and a peer's reward for the same option—the system can safely translate external peer data into a local valuation scale.

This mechanism is highly effective for crowdsourced environments. As Yang Liu et al. show in Crowd Learning: Improving Online Decision Making Using Crowdsourced Data, modeling subjective valuation scales as individual distortion factors allows a system to integrate heterogeneous feedback (such as correcting for a user whose "3-star" rating is equivalent to another's "5-star" rating) without skewing the collective model.

Mechanism 3: Bypassing Utilities via Directional Oracles

The most radical solution to the scale misalignment problem is to abandon numerical utility values entirely. If the absolute numbers are the source of the bias, we can design mechanisms that rely strictly on ordinal or directional feedback.

This is the intuition behind the Direction-based Bargaining Solution (DiBS), proposed by Kushagra Gupta et al. in Cooperative Bargaining Games Without Utilities: Mediated Solutions from Direction Oracles. Instead of asking agents to report a numerical payoff value, a central mediator queries "direction oracles," which simply indicate which way an agent wants to move within the decision space.

By stripping away the magnitude of the response and focusing purely on the geometry of preference, the authors demonstrate that a mediator can guide agents to a fair resource allocation that is uniquely robust to non-linear scaling. This completely neutralizes the incentive for strategic agents to lie or exaggerate their internal metrics, offering a elegant path forward for cross-agent value grounding.

Key Foundational Work

Моральные основания политики (2003) — contributed 2 concept elements to this direction.
Deals Among Rational Agents (1985) — contributed 1 concept elements to this direction.
On the Foundations of Expected Expected Utility (2003) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_interpersonal_utility_comparison_e86cd825,
  title = {interpersonal utility comparison},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/interpersonal-utility-comparison/rdr_e86cd8254b8c461eae4b1353e6485d70},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_interpersonal_utility_comparison_e86cd825_v1,
  title = {interpersonal utility comparison},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/interpersonal-utility-comparison/rdr_e86cd8254b8c461eae4b1353e6485d70/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake