In-Sample Value Estimation via Expectile Regression

This direction is merged into another direction: extrapolation error (out-of-distribution value overestimation). Its last version stays readable and citable here.

Version 1 of 130 papers34 conceptsfrozen 2026-10-04

In offline reinforcement learning (RL), we face a fundamental tension: we want to find a policy that is better than the one that collected our data, but we cannot interact with the environment to verify if our "better" ideas actually work. When we use standard temporal difference (TD) learning, we typically use a maximum operator over actions: $V(s) = \max_a Q(s, a)$. In an offline setting, this is catastrophic. The model "hallucinates" high values for out-of-distribution (OOD) actions, leading to massive overestimation bias that collapses the learning process.

To solve this, a growing body of research has converged on a shared mechanism: expectile regression. This technique allows agents to estimate the upper bounds of achievable returns while strictly staying "in-sample"—meaning they never have to query the value of an action they haven't actually seen.

The Intuition of the Asymmetric Loss

Standard mean squared error (MSE) treats all residuals equally, finding the mean of a distribution. Expectile regression introduces an asymmetry. By weighting positive residuals more heavily than negative ones (using a parameter $\tau > 0.5$), the model stops trying to find the average value and starts trying to find a specific quantile-like value in the upper tail of the distribution.

As Charlie Snell and Ilya Kostrikov demonstrate in Implicit Language Q Learning, this allows the value function to implicitly approximate a maximum over the actions present in the dataset. If $\tau$ is high, the value function $V(s)$ tracks the best outcomes observed at state $s$. Because this estimation only uses transitions already in the replay buffer, it avoids the "OOD trap" entirely.

Stitching Suboptimal Trajectories

One of the most powerful properties of RL is "stitching": the ability to take the first half of a good trajectory and the second half of another to create a new, optimal path. Sequence models like the Decision Transformer often struggle with this because they treat RL as a pure supervised prediction task.

Zifeng Zhuang and Jinxin Liu address this in Reinformer by using expectile regression to predict the "max-return" achievable from any state. By conditioning a Transformer on these expectile-based value estimates rather than simple "return-to-go" labels, the model learns to navigate toward high-value states it has seen elsewhere in the dataset. Similarly, Chenxiao Gao in ACT uses expectiles to generate advantage labels, providing a stable signal for sequence models to "stitch" together optimal segments without querying impossible actions.

Decoupling Actions from Values

Traditional RL couples the value of a state to the specific action taken. However, in complex or stochastic environments, this coupling can be noisy. Jiafei Lyu and Zongqing Lu propose State Advantage Weighting (SAW), which uses expectile regression to learn state-state value functions ($Q(s, s')$). This decouples "where to go" from "how to get there," allowing the agent to focus on reaching high-value future states identified through the upper-tail of the reward distribution.

This "best-case" estimation is also vital for safety. In C2IQL, Zifan LIU and Jun Zhang use expectiles to estimate the best possible rewards within a safety budget. By staying in-sample, the agent avoids overestimating the safety of unknown actions, which is a common cause of catastrophic failure in safe RL.

Solving the Long Horizon

Standard TD learning suffers from "bias propagation"—errors in value estimation at step $T$ compound as they are bootstrapped back to step 0. Seohong Park and Sergey Levine tackle this in Transitive RL and Decoupled Q-Chunking.

Instead of step-by-step updates, they use a divide-and-conquer approach. Expectile regression acts as a "soft-max" over potential intermediate subgoals. This allows the model to verify if a long-range path is feasible by checking if high-value intermediate states exist in the data, reducing the number of recursive updates from $O(T)$ to $O(\log T)$.

Seizing Serendipity

While most work uses expectiles to prevent overestimation, Tianying Ji and Fuchun Sun argue in Seizing Serendipity that expectiles can also solve underestimation. In the later stages of training, standard actors might become too conservative. By using expectile regression to identify the "serendipitous" high-performing actions hidden in the replay buffer, the BEE operator allows the agent to exploit past successes that the current policy might have otherwise ignored.

By replacing the explicit $\max$ with an implicit expectile, these researchers have turned a dangerous optimization problem into a stable statistical estimation problem, enabling RL to function reliably on static, real-world datasets.

Papers in this version

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_in_sample_value_estimation_via_expectile_12b2ef61,
  title = {In-Sample Value Estimation via Expectile Regression},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/in-sample-value-estimation-via-expectile-regression/rdr_12b2ef6124b549aa9603882d2ddd9025},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_in_sample_value_estimation_via_expectile_12b2ef61_v1,
  title = {In-Sample Value Estimation via Expectile Regression},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/in-sample-value-estimation-via-expectile-regression/rdr_12b2ef6124b549aa9603882d2ddd9025/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}