Beyond the Linear Assumption in Reward Modeling

891 concepts657 papers6 proposals

In sequential decision-making frameworks—ranging from contextual bandits to reinforcement learning from human feedback (RLHF)—the assumption of linear realizability has long been the default paradigm. This assumption posits that the reward or utility of an action can be modeled as a simple inner product between a known feature vector and an unknown parameter vector.

While mathematically convenient, this linear assumption frequently fails when confronted with the complexity of real-world environments and human subjectivity. A growing body of research focuses on breaking free from this linear constraint, exploring how and why non-linear preference and reward modeling fundamentally changes an agent's ability to navigate complex trade-offs.

The Convenience and Failure of Linear Realizability

To understand why non-linear modeling is necessary, we must first understand why the linear assumption is so pervasive. In contextual bandits, assuming a linear relationship allows algorithms to construct elegant geometric confidence ellipsoids around parameter estimates. For instance, the foundational OFUL algorithm by Yasin Abbasi-Yadkori et al. and the contextual Thompson Sampling framework by Shipra Agrawal et al. rely entirely on this linear structure to prove strong, high-probability bounds on cumulative regret.

However, real-world data rarely adheres to strict linearity. In personalization systems, such as the news recommendation engine developed by Lihong Li et al., assuming a linear relationship between user context and engagement overlooks complex behavioral patterns.

Consider a recommendation scenario: a user might enjoy both "long-form investigative journalism" and "breaking tech news," but their interest in a "long-form breaking tech article" is not a simple additive sum of those two features. Features interact in highly non-linear, synergistic, or antagonistic ways that a linear model fundamentally cannot capture.

The Hidden Costs of Model Misspecification

When an algorithm forces a linear model onto a non-linear reward landscape, it suffers from model misspecification. This mismatch introduces several severe consequences:

  1. Extensive Manual Feature Engineering: As noted by the authors of A Practical Method for Solving Contextual Bandit Problems Using Decision Trees, linear models require human experts to manually construct cross-features and non-linear transformations to align the input data with the model's rigid parametric assumptions.
  2. Breakdown of Dimensionality Reduction: Dimensionality reduction techniques like random projections, which are highly effective for linear payoffs, become bottlenecked. Michael Lyu et al. point out that relying on linear payoff structures restricts these frameworks to environments where rewards can be modeled purely as inner products, rendering them ineffective for black-box or intricate reward functions.
  3. Poor Generalization and Transfer: In Inverse Reinforcement Learning (IRL), the goal is to recover an underlying reward function from expert demonstrations. The authors of Nonlinear Inverse Reinforcement Learning with Gaussian Processes demonstrated that linear reward models suffer massive performance degradation when an agent is transferred to an entirely new environment. Non-linear models, by contrast, capture the true underlying structure of the task, maintaining stable error rates across environment shifts.

The Multi-Objective Trap: Why Weighted Sums Fail Human Alignment

The limitations of linearity become acute when aligning artificial intelligence with human preferences. A common shortcut in multi-objective reinforcement learning is to scalarize a set of distinct objectives (such as helpfulness, safety, and conciseness) by taking a weighted linear combination:

$$R_{\text{total}} = w_1 \cdot R_{\text{helpful}} + w_2 \cdot R_{\text{safe}}$$

This formulation assumes that human values operate like a constant-ratio barter system. But human psychology does not work this way. Zhanhui Zhou et al. argue that competing objectives exhibit non-linear interactions where trade-offs do not follow a constant ratio. A massive increase in helpfulness cannot linearly compensate for a catastrophic drop in safety.

This linear multi-objective trap is a recurring theme across preference elicitation frameworks:

  • Minyoung Hwang et al. demonstrate that a simple weighted sum fails to capture the intricate trade-offs inherent in personalizing robotic behaviors.
  • Alexandre Rame et al. show that weight-averaging models fine-tuned on diverse rewards fail to represent sophisticated human judgments because they assume a linear preference space.
  • Jessy Lin et al. emphasize that when inferring rewards from natural language, the value of one attribute depends non-linearly on the state of another, creating conditional dependencies that linear models completely miss.

Mathematical Frameworks for Non-Linear Rewards

To preserve the theoretical guarantees of sequential decision-making while embracing non-linear realities, researchers have advanced several mathematical paradigms:

Generalized Linear Models (GLMs)

In many practical settings, rewards are binary (e.g., a user either clicks an article or does not). This necessitates logistic regression or other GLMs where the expected reward is filtered through a non-linear link function. Lihong Li et al. closed a major theoretical gap by establishing sharp, finite-sample confidence bounds for maximum likelihood estimates in GLMs, proving that non-linear models can achieve minimax-optimal regret guarantees comparable to their linear counterparts.

Kernel Methods

Instead of explicitly engineering non-linear features, kernel methods implicitly map contexts into high-dimensional reproducing kernel Hilbert spaces (RKHS). Michal Valko et al. introduced KernelUCB, an algorithm that models non-linear relationships between actions and rewards using kernels. They proved that its cumulative regret scales with the "effective dimension" of the data, matching the lower bounds of linear bandits without requiring linear realizability.

Logical and Structural Dependencies

Expert behavior is often governed by logical rules rather than smooth mathematical functions. Sergey Levine et al. developed FIRL (Feature construction for IRL), which uses an iterative regression tree approach to automatically construct non-linear reward features. This allows the system to capture conditional dependencies—such as an agent understanding that "speeding" is acceptable only if it is combined with the condition "driving an ambulance."

Modern Frontiers in Non-Linear Preference Modeling

As the field shifts toward aligning large language models and complex autonomous agents, non-linear preference modeling has transitioned from a theoretical luxury to an algorithmic necessity.

When aggregating preferences across a population, Jessica Dai et al. highlight that extending democratic social choice axioms to deep neural networks requires moving past linear score aggregations. Similarly, Lirong Xia argues that current preference models fail to capture the non-linear trade-offs inherent in human concepts of fairness and simplicity, proposing evolutionary policy architectures to optimize for these non-linear structures directly.

To handle these non-linear spaces at scale, researchers are deploying generative and graph-based architectures:

  • Diffusion Models: In offline multi-objective RL, Yifu Yuan et al. introduced MODULI, which uses diffusion models to generate trajectories across a non-linear preference space, successfully generalizing to out-of-distribution human preferences and mapping complex Pareto fronts.
  • Graph Learning: For multi-turn user interactions, Encheng Xie et al. developed GMTRouter, which models user-AI interactions as a heterogeneous graph. This approach bypasses linear utility functions entirely, capturing the nuanced, qualitative factors that dictate human subjectivity over time.

By moving beyond the linear reward assumption, these diverse frameworks allow machine learning systems to stop optimizing for mathematically convenient proxies and start optimizing for the true, non-linear complexity of the real world.

History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_non_linear_preference_modeling_6c63d107,
  title = {non-linear preference modeling},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/non-linear-preference-modeling/rdr_6c63d10749854684872e9e11e83b0680},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_non_linear_preference_modeling_6c63d107_v1,
  title = {non-linear preference modeling},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/non-linear-preference-modeling/rdr_6c63d10749854684872e9e11e83b0680/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake