When we evaluate an AI's response, we often compress our judgment into a single metric: a thumbs-up, a five-star rating, or a scalar reward score. This mathematical convenience underpins traditional Reinforcement Learning from Human Feedback (RLHF). By reducing human preference to a single real number, optimization becomes straightforward.
However, human values are inherently multi-dimensional. A response can be factually flawless but dry, or highly creative but slightly inaccurate. When we force an alignment algorithm to compress these distinct axes into a single scalar, we create a severe information bottleneck. A growing body of research demonstrates that this "scalar reward reductionism" is not just theoretically limiting—it is the root cause of major alignment failures, from reward hacking to rigid, unsteerable models.
The Information Bottleneck of Scalar Rewards
Why is a single number insufficient? As Peter Vamplew et al. argue in Scalar reward is not enough: a response to Silver, Singh, Precup and Sutton (2021), reducing complex, multi-faceted human objectives to a single scalar value treats a narrow subset of utility functions as a general case. This mathematical oversimplification prevents an agent from representing nuanced preferences, such as lexicographic ordering (where one condition must be strictly satisfied before another is considered).
In Beyond Reward: Offline Preference-guided Policy Optimization, Li He formalizes this limitation as an information bottleneck. When multi-dimensional feedback is compressed into a single scalar, the underlying structural reasons why a human preferred one behavior over another are discarded. The learning agent receives a signal that a response is "good," but it cannot discern whether it was chosen for its accuracy, its tone, or its formatting.
Scalar Reward Approach:
[Nuanced Human Feedback] ---> (Compression Bottleneck) ---> [Single Scalar: 0.85] ---> Loss of "Why"
Multi-Dimensional Approach:
/---> Helpfulness: 0.92
[Nuanced Human Feedback] ---+----> Harmlessness: 0.78
\---> Conciseness: 0.45
This lack of granularity manifests in several distinct failure modes during model training:
- Reward Hacking via Superficial Traits: Without explicit dimensions, models find the path of least resistance to maximize the scalar score. As Vijay Viswanathan et al. demonstrate in Checklists Are Better Than Reward Models For Aligning Language Models, scalar reward models frequently prioritize superficial qualities like length and politeness over strict adherence to complex instructions.
- The "Average Preference" Trap: Human preferences are diverse and often contradictory. Haoxiang Wang et al. point out in Arithmetic Control of LLMs for Diverse User Preferences: Directional Preference Alignment with Multi-Objective Rewards that collapsing these variations into a single scalar forces the model to target a generic, averaged preference, failing to accommodate individual user requirements.
- Incommensurable Trade-offs: In high-stakes dimensions like safety versus utility, scalar aggregation creates dangerous compromises. In Safe RLHF: Safe Reinforcement Learning from Human Feedback, Josef Dai et al. explain that combining helpfulness and harmlessness into a single scalar creates an artificial tension, preventing the model from maximizing both utility and safety simultaneously.
Decomposing the Preference Space
To break the bottleneck, researchers are shifting from scalar rewards to multi-dimensional vector spaces. The intuition is simple: if human judgment consists of multiple independent axes, the reward signal should mirror that geometry.
One approach is to mathematically extract these hidden dimensions from standard preference data. In Rethinking Diverse Human Preference Learning through Principal Component Analysis, Feng Luo et al. introduce Decomposed Reward Models (DRMs). By applying Principal Component Analysis (PCA) to the differences between preferred and rejected responses, the authors isolate distinct, orthogonal basis vectors representing attributes like factual accuracy and conversational politeness—all without requiring expensive multi-aspect labels.
Similarly, Zhuocheng Gong et al. propose Latent Preference Coding: Aligning Large Language Models via Discrete Latent Codes, which models human judgment as a mixture of discrete latent factors, allowing standard alignment algorithms to balance competing objectives automatically.
Scalar Optimization: Maximize E[R(x)] -> Single compromised solution
Pareto Optimization: Optimize [R_help(x), R_safe(x), R_brief(x)] -> Discovers the entire trade-off frontier
When these dimensions are explicitly separated, alignment transforms from a simple maximization problem into a multi-objective optimization problem. Instead of seeking a single optimal point, models can map out a Pareto frontier—a set of solutions where no single attribute can be improved without degrading another.
In Panacea: Pareto Alignment via Preference Adaptation for LLMs, Huifang Chen et al. exploit this by injecting a dynamic preference vector into a model's weights via low-rank adaptation (LoRA). This allows a single model to traverse the entire Pareto front at inference time based on user demand. A similar multi-aspect Pareto framework is applied to recommendation systems by Xu Chen et al. in Reinforcement Recommendation with User Multi-aspect Preference, balancing conflicting item features like price and location without introducing gradient bias.
Replacing Numbers with Language
Perhaps the most radical departure from scalar reductionism is abandoning numerical rewards entirely, replacing them with text tokens or natural language feedback. If human preferences are best expressed through language, then the reward model should output text.
Yi Dong et al. pioneer this with SteerLM in SteerLM: Attribute Conditioned SFT as an (User-Steerable) Alternative to RLHF. Instead of using an RL loop guided by a scalar score, they condition Supervised Fine-Tuning (SFT) on explicit attribute tokens (such as <Helpfulness:5> <Humor:3>). At inference time, the user simply types the desired tokens to arithmetically steer the model's behavior.
Taking this a step further, Jiayi Zhou et al. introduce sequence-to-sequence reward modeling in Sequence to Sequence Reward Modeling: Improving RLHF by Language Feedback.
Standard Reward Model:
[Prompt + Response] ---> [Neural Network] ---> Scalar Value (e.g., -1.4)
Seq2Seq Reward Model:
[Prompt + Response] ---> [Neural Network] ---> "The response is polite but factually incorrect in paragraph 2."
By training the reward model to generate token-by-token textual corrections rather than a single number, the authors completely bypass the scalar bottleneck. The language model learns why a response is flawed, allowing for precise, granular adjustments that mitigate common alignment issues like over-refusal and length bias.
Scaling Multi-Dimensional Alignment
As the field moves away from scalar reductionism, the engineering challenge shifts from stabilizing volatile RL loops to managing multi-task interference. When a model tries to learn dozens of distinct preference dimensions simultaneously, the gradients can conflict, causing the model to forget previously learned traits.
To address this, Shanghaoran Quan introduces a Double-Layer Mixture-of-Experts architecture in DMoERM: Recipes of Mixture-of-Experts for Effective Reward Modeling. By routing inputs to task-specific experts and further decomposing them into fine-grained capability dimensions, the architecture prevents different values from diluting one another.
Concurrently, Xingzhou Lou et al. develop Sequential Preference Optimization: Multi-Dimensional Preference Alignment with Implicit Reward Modeling, a framework that trains models on new preference dimensions sequentially using mathematical constraints to ensure that new attributes are acquired without degrading performance on prior ones.
Ultimately, breaking the scalar reward bottleneck changes how we conceptualize AI alignment. Alignment is no longer viewed as maximizing a single, elusive "human value" score. Instead, it is treated as the deliberate navigation of a dense, multi-dimensional landscape of human preferences, giving models the granularity they need to handle the real world's trade-offs.
Key Foundational Work
Reinforcement Recommendation with User Multi-aspect Preference (2021) — contributed 1 concept elements to this direction.
Multi-Objective SPIBB: Seldonian Offline Policy Improvement with Safety Constraints in Finite MDPs (2021) — contributed 1 concept elements to this direction.
Scalar reward is not enough: a response to Silver, Singh, Precup and Sutton (2021) (2022) — contributed 1 concept elements to this direction.