Expected Scalarised Returns for Single Execution Decision Making

18 concepts13 papers1 proposal

Standard reinforcement learning (RL) operates on a statistical luxury: the law of large numbers. Most agents are trained to maximize the Scalarised Expected Return (SER), which optimizes the utility of the average outcome across many repeated trials. This works well for fleet management or warehouse robotics, where a poor outcome in one run is mitigated by thousands of successful ones.

However, in high-stakes domains like medical emergency response or infrastructure maintenance, we often only get one shot. As Jiachen Yang et al. argue, a medical AI cannot afford a "training phase" of failed treatments on a unique patient; it must succeed in its very first interaction. In these "one-shot" scenarios, the optimization objective must shift from the utility of the average to the Expected Scalarised Return (ESR)—the average utility of individual, non-repeatable trajectories.

The Distinction Between Average and Individual Outcomes

The core difference between SER and ESR lies in when the "scalarisation" (the application of a utility function to multiple objectives) occurs. Roxana Rădulescu and Patrick Mannion provide a taxonomy that clarifies this:

  • SER (Scalarised Expected Return): You average the rewards over many runs first, then apply the utility function. This is suitable when the user cares about the total accumulated performance over time.
  • ESR (Expected Scalarised Return): You apply the utility function to the outcome of each individual run first, then average those utilities. This is essential when the outcome of a single execution is the primary concern.

This distinction is mathematically trivial if the utility function is linear, but as Roxana Radulescu et al. point out, real-world preferences are rarely linear. When utility functions are non-linear—such as when there are strict safety thresholds or diminishing returns—a policy that is optimal on average (SER) might produce catastrophic results in every single execution.

Why the "Average" Fails in High-Stakes Scenarios

Consider the maintenance of historic quay walls, as studied by Jesse van Remmerden et al.. If an agent optimizes for the average cost and safety across a century, it might accept a high risk of collapse in one year because the "average" safety remains high. However, a single collapse is irreversible. By using ESR, the agent prioritizes the utility of specific execution paths, ensuring that safety thresholds are respected across all possible scenarios, rather than just on average.

This shift is particularly critical when dealing with risk. Conor F. Hayes et al. note that SER assumes performance can be averaged over infinite trials to mitigate risk. In one-shot tasks, this assumption is invalid. An agent must instead account for the full distribution of potential returns to avoid a single catastrophic failure.

Algorithms for One-Shot Success

To optimize for ESR, agents cannot simply track mean values; they must understand the probability distribution of their rewards. Several recent approaches have been developed to handle this:

  1. Distributional Planning: Conor F. Hayes et al. introduced Multi-Objective Distributional Value Iteration (MODVI), which maintains the full distribution of potential returns rather than just the expected value. This allows the agent to plan for the best expected utility in a single-execution scenario.
  2. Distributional MCTS: For complex environments, Enda Howley et al. and Conor F. Hayes et al. developed Monte Carlo Tree Search (MCTS) variants that use distributional methods like Thompson Sampling. These algorithms evaluate the utility of individual paths during the search process, making them inherently risk-aware.
  3. Single Episode Policy Transfer (SEPT): In environments with unknown dynamics, Brenden K. Petersen and Daniel Faissol propose using a brief "probing" phase at the start of an episode. This allows the agent to identify latent environmental features and adapt its policy immediately, achieving near-optimal performance without needing multiple trial-and-error rollouts.
  4. Occupancy MDPs: Pedro Pinto Santos et al. frame the single-trial regime as an "Occupancy MDP," where history-dependent planning allows the agent to adapt to the specific stochastic outcomes it encounters in real-time, rather than following a fixed policy that was only optimal on average.

Dominance in the ESR Space

When a user's specific utility function is unknown, the problem becomes finding a set of policies that are "optimal" under any possible preference. Timothy Verstraeten et al. introduced the ESR set, a new solution concept based on stochastic dominance. Unlike the standard Pareto front used in SER, the ESR set identifies policies that are likely to provide the highest utility for a single execution, regardless of the specific non-linear trade-offs the user prefers.

By shifting the focus from aggregate statistics to trajectory-level utility, these methods bridge the gap between theoretical reinforcement learning and the uncompromising requirements of real-world, one-shot decision making.

Key Foundational Work

Recent Advances in Multi-Objective Multi-Agent Decision Making (2020) — contributed 1 concept elements to this direction.
Single Episode Policy Transfer in Reinforcement Learning (2020) — contributed 1 concept elements to this direction.
Single Episode Policy Transfer in Reinforcement Learning (2019) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_expected_scalarised_returns_c3dfe9d7,
  title = {expected scalarised returns},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/expected-scalarised-returns/rdr_c3dfe9d7b8584f99850af91c562e2058},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_expected_scalarised_returns_c3dfe9d7_v1,
  title = {expected scalarised returns},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/expected-scalarised-returns/rdr_c3dfe9d7b8584f99850af91c562e2058/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake