Optimistic Exploration under Pessimistic Safety Constraints

This direction is merged into another direction: safe exploration, optimistic-pessimistic exploration. Its last version stays readable and citable here.

Version 1 of 1195 papers257 conceptsfrozen 2026-10-04

In traditional optimization and reinforcement learning, the "Principle of Optimism in the Face of Uncertainty" is the standard engine for discovery. If an agent doesn't know how rewarding an action is, it assumes the best-case scenario. This "optimism" encourages the agent to try new things, eventually learning the true value of every option.

However, in real-world engineering—from tuning a power grid to training a robot—discovery has a dangerous flip side: unknown constraints. If you are optimistic about a system's limits, you might accidentally trigger a catastrophic failure. This research direction explores how to balance the "gas pedal" of optimistic exploration with the "brakes" of pessimistic safety.

The Tension of Two Uncertainties

When optimizing a "black-box" function where the rules are as unknown as the rewards, an agent faces two distinct types of uncertainty. Spencer Hutchinson and Berkay Turan note in Exploiting Problem Geometry in Safe Linear Bandits that failing to distinguish between these leads to inefficient sampling.

  1. Reward Uncertainty: "I don't know how much gold is in that cave."
  2. Constraint Uncertainty: "I don't know if the cave ceiling will collapse if I enter."

If the agent is pessimistic about both, it will never leave the entrance. If it is optimistic about both, it might find the gold but perish in a cave-in. The core insight of this field is to decouple these uncertainties, applying optimism to rewards to drive progress and pessimism to constraints to maintain safety.

Optimistic Pessimism: The Dual-Track Approach

Several researchers, including Ruida Zhou and Dileep Kalathil, have formalized this as "Optimistic Pessimism." In Learning Policies with Zero or Bounded Constraint Violation for Constrained MDPs, they demonstrate that an agent can achieve optimal learning rates by maintaining two different confidence bounds:

  • Upper Confidence Bounds (UCB) for Rewards: The agent acts as if the reward is the highest plausible value within its current knowledge.
  • Lower Confidence Bounds (LCB) for Safety: The agent acts as if the risk is the worst plausible value.

This asymmetry is the key to "Safe Exploration." As Yarden As and Andreas Krause describe in ActSafe, epistemic uncertainty (what the model doesn't know) acts as an "intrinsic reward" to encourage exploration, but that same uncertainty triggers conservative behavior when it pertains to safety costs.

A major challenge in this area is that the most "optimal" solutions often lie exactly on the edge of what is allowed. To find the global optimum, an algorithm must aggressively explore the boundaries of the feasible region.

In the CONFIG algorithm, Wenjie Xu and Colin Jones take a slightly more aggressive stance. In Constrained Efficient Global Optimization of Expensive Black-box Functions, they apply optimism to both the objective and the constraints. By treating a point as feasible if there is even a possibility it satisfies the constraint, CONFIG can "peek" over the fence to see if a better region exists. This is particularly useful when the agent hasn't found any safe starting point yet, a problem also addressed by Shion Takeno in Sequential and Parallel Constrained Max-value Entropy Search.

Conversely, for high-stakes environments where a single violation is a "dealbreaker," algorithms like OPSE-LCMDP by Toshinori Kitamura and colleagues guarantee zero safety violations during the entire learning process. They achieve this by ensuring exploration only occurs when the agent is statistically certain that the safety constraints will not be violated.

Decoupling Direction and Magnitude

A recent refinement in this field involves decomposing the action itself. In Exploiting Problem Geometry in Safe Linear Bandits, Spencer Hutchinson introduces Directional Optimism.

Instead of being generally cautious, the agent uses optimism to pick a direction (which way looks most promising?) and then uses pessimism to scale the magnitude (how far can I safely go in that direction?). This allows for aggressive exploration of high-reward directions while maintaining a strict safety buffer, significantly reducing the "cost of dimensionality" in complex problems.

From Online to Offline Safety

The same principles are being applied to "Offline Reinforcement Learning," where an agent learns from a static dataset without interacting with the environment. In this setting, the risk is "overoptimization"—targeting a reward peak that only exists because of noise in the data.

Frameworks like VOCE and POCE, developed by Jiayi Guan and Jiaming Ji, use conservative estimation to penalize rewards in regions where data is sparse. By being pessimistic about both rewards and costs in "out-of-distribution" areas, these agents learn to stay within the "safe envelope" of the original dataset while still finding the best possible policy within those bounds.

Papers in this version

Showing 60 of 195 papers.

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_optimistic_exploration_under_pessimistic_04c4354e,
  title = {Optimistic Exploration under Pessimistic Safety Constraints},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/optimistic-exploration-under-pessimistic-safety-constraints/rdr_04c4354e6628411c9a1a003ea4022985},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_optimistic_exploration_under_pessimistic_04c4354e_v1,
  title = {Optimistic Exploration under Pessimistic Safety Constraints},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/optimistic-exploration-under-pessimistic-safety-constraints/rdr_04c4354e6628411c9a1a003ea4022985/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}