Aligning AI with Latent Human Reward Functions

This direction is merged into another direction: inverse reinforcement learning. Its last version stays readable and citable here.

Version 1 of 115 papers17 conceptsfrozen 2026-10-04

When we interact with digital systems, our actions are rarely the result of a single, simple goal. A user clicking a video might be seeking entertainment, but they might also be satisfying a fleeting curiosity or even clicking by mistake. Traditional AI alignment often relies on surface-level metrics—like click-through rates or explicit "constitutions"—to guide behavior. However, these proxies frequently fail to capture the nuanced, often implicit, values that drive human decision-making.

A growing body of research suggests a more robust path: treating human behavior as a demonstration of an underlying "policy" and using Inverse Reinforcement Learning (IRL) to reverse-engineer the latent reward functions that make that behavior appear optimal.

The Inverse Problem of Human Intent

In standard Reinforcement Learning, we provide an agent with a reward function, and it learns a policy (a way of acting) to maximize that reward. Inverse Reinforcement Learning flips this script. It assumes the human is the agent, and their observed actions are the "optimal" output of some internal, hidden reward function.

By observing sequences of interactions, researchers like Ziming Li and Maarten de Rijke argue that we can "reverse engineer" these internal weights. In Towards Learning Reward Functions from User Interactions, they demonstrate that interaction logs—the breadcrumbs of search queries and clicks—contain enough signal to recover personalized preferences that static metrics miss. This allows a system to understand not just what a user did, but the "why" behind the sequence.

Beyond Clicks to Mental Rewards

The shift from surface actions to latent intent is most visible in recommendation systems. Standard models optimize for immediate engagement, which can lead to "clickbait" loops. Mengchen Zhao and colleagues introduced MTRec in Learning to Align with User Preferences via Mental Reward Models to solve this.

Instead of treating a "view" as the ultimate goal, MTRec models the user as an agent with a "mental reward" for satisfaction. By applying IRL to viewing patterns on short-video platforms, the system learns to prioritize long-term satisfaction over short-term clicks, resulting in significant gains in actual user viewing time. The intuition is simple: if we can model the user's internal satisfaction as a mathematical function, we can optimize the AI to serve that function directly.

Inferring Social and Moral Contracts

This approach scales from individual preferences to complex social norms. Humans rarely articulate the "social contracts" they follow, yet they expect AI to adhere to them.

In Social Contract AI: Aligning AI Assistants with Implicit Group Norms, Sam Kwok and Alex Tamkin demonstrate that AI can infer group norms by observing how people interact in economic games. Rather than being told "be fair," the AI observes fair behavior and infers that "fairness" is a high-weight component of the group's reward function.

Similarly, Nigini Oliveira and Andrew N. Meltzoff show in Culturally-Attuned Moral Machines that IRL can help AI "absorb" cultural values like altruism. By analyzing action traces from different communities, the AI learns to apply these implicit values to entirely new scenarios, effectively learning a "moral compass" from observation rather than instruction.

Real-Time Adaptation and Simulation

The utility of latent reward inference extends into high-stakes physical interactions and system design:

  • Shared Control: In autonomous driving, Haoran Wang and Jia Hu use IRL to identify a driver's specific preferences—such as their preferred following distance or acceleration comfort—in real-time. This allows the vehicle to smoothly transition control during handovers by matching the machine's behavior to the driver's latent style.
  • User Simulation: To train conversational agents without exhausting real humans, researchers like Laure Soulier use IRL to build synthetic user simulators. These simulators don't just mimic text; they mimic the underlying motivations of a human searcher, providing a more realistic training ground for RL agents.
  • Behavioral Analysis: Even on platforms like Reddit, IRL reveals hidden structures. Lanqin Yuan and Marian-Andrei Rizoiu found that users in vastly different subreddits often share identical behavioral policies, suggesting that our "interaction style" is a latent trait that transcends the topics we discuss.

Why It Works

The core intuition behind this research direction is that behavior is a high-bandwidth signal of intent. While language can be ambiguous and explicit rewards (like likes) are easily "gamed," the consistent choices a human makes over time reveal a stable underlying logic. By framing alignment as an inverse problem, we move from asking AI to follow our orders to asking AI to understand our values. This creates systems that are not just more accurate, but more "attuned" to the subtle, unwritten rules of human life.

Papers in this version

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_aligning_ai_with_latent_human_reward_fun_d1da73a7,
  title = {Aligning AI with Latent Human Reward Functions},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/aligning-ai-with-latent-human-reward-functions/rdr_d1da73a712794cd08e78f42f3fddff71},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_aligning_ai_with_latent_human_reward_fun_d1da73a7_v1,
  title = {Aligning AI with Latent Human Reward Functions},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/aligning-ai-with-latent-human-reward-functions/rdr_d1da73a712794cd08e78f42f3fddff71/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}