Future-Informed Actor-Critic

Samy Mokeddem, Gaspard Lambrechts, Adrien Bolland, Damien ErnstarXiv 2026 · arXiv

TL;DR

This paper introduces Future-Informed Actor-Critic (FIAC), a reinforcement learning framework that conditions the critic on "privileged" information from the future—specifically states, actions, and rewards that occur after the current timestep. By incorporating endogenous features (those dependent on the agent's own actions), FIAC significantly improves credit assignment and performance in environments where rewards are sparse or corrupted by noise.

What Problem Does This Paper Address?

In standard Actor-Critic (AC) methods, the critic's job is to estimate the value of a state or action to help the actor improve. Usually, this critic only sees the current state or history. However, in many complex tasks, the consequences of an action aren't immediate. This leads to the credit assignment problem: determining which past action was actually responsible for a reward received much later.

While some "asymmetric" AC methods use privileged information during training, they typically restrict this to exogenous features—environmental factors independent of the agent's actions—to ensure the policy gradient remains unbiased. The authors argue that this overlooks endogenous features (the agent's own future actions and their consequences), which are arguably the most critical signals for solving the credit assignment puzzle.

Methods

The authors bridge this gap by introducing the future-informed value function ($F^\pi$), which is conditioned on a feature $z_t$ derived from a window of the future trajectory.

Key technical contributions include:

  • Future-Informed Policy Gradient (FIPG): A theoretical proof showing that the standard policy gradient can be exactly expressed using this future-informed critic.
  • Contraction Mapping: They prove that $F^\pi$ is the unique fixed point of a $\gamma$-contractive operator (under certain conditional independence assumptions), which justifies using standard Temporal Difference (TD) learning to train the future-informed critic.
  • The FIAC Algorithm: During training, the critic $F_\phi(s, a, z)$ is fed a window of future states and actions (e.g., the next 20 steps). Because the critic is only used to compute gradients during training and is discarded at test time, the agent doesn't need to "see the future" to act in the real world.

Datasets

The researchers evaluated FIAC on several environments designed to test credit assignment and robustness to noise:

  • Umbrella Chain: A diagnostic task where only the very first action determines the final reward, but the agent receives distracting random rewards at every intermediate step.
  • FrozenLake (8x8): A grid-world with stochastic transitions, tested in both standard and "Noisy" variants (where Gaussian noise is added to every reward).
  • CartPole: A classic dense-reward control task, also tested with a "Noisy" variant.

Key Insights and Ideas

  • The Power of Endogenous Info: The most striking "aha" moment is that knowing what the agent did in the future is more helpful for the critic than just knowing what the environment did. In the Umbrella Chain task, conditioning on the terminal reward (an endogenous feature at $t=0$) was the only way to consistently solve the problem.
  • Future States are King: Ablation studies (see Figure 2) revealed that future states provide the strongest signal for performance gains. Interestingly, adding future rewards to the critic's input actually degraded performance in noisy environments, likely because the rewards themselves were the source of the noise.
  • Sparse/Noisy Specialist: FIAC isn't a "silver bullet" for every task. In dense-reward environments like standard CartPole, it matches baseline performance but converges more slowly. Its true strength lies in "dirty" environments where the reward signal is weak or obscured.

Key Results

FIAC demonstrated superior performance in challenging scenarios compared to standard baselines like A2C and TD-based Actor-Critic:

  • Noisy FrozenLake: FIAC achieved significantly higher expected returns than all baselines, which struggled to distinguish the true reward signal from the Gaussian noise.
  • Umbrella Chain: While standard algorithms failed to solve the long-term credit assignment, FIAC with terminal reward features reached near-perfect returns.
  • Ablation Performance: As shown in the figure below, the choice of future features is critical. The version using both future states and actions ($z = (s_{t:20}, a_{t:20})$) yielded the highest returns with the lowest variance.
Figure 1: Evolution of the expected return of the studied algorithms in each environment. The expected return is approximated by the interquartile mean (IQM) over 10 seeds and 90% confidence intervals are reported, following Agarwal et al. (2021).

Limitations and Future Work

  • Computational Overhead: Conditioning the critic on a window of future trajectories increases the input dimensionality and computational cost during training.
  • Convergence Speed: In simple, dense-reward tasks, the added complexity of the future-informed critic can lead to slower convergence compared to simpler methods.
  • Underlying Mechanism: While the empirical results are strong, the authors note it remains unclear exactly why this helps—whether it's primarily due to better representation learning, a more favorable bias-variance tradeoff, or a different learning phenomenon entirely.
  • Scaling: Future work should explore how this scales to high-dimensional continuous control tasks and more complex "real-world" simulators.

Summary updated

Abstract

Actor-critic methods optimize a parametrized policy using a critic to estimate their policy gradient. Recent work has shown that conditioning the critic on features of future states, actions, or rewards can improve sample efficiency and even final return. However, existing methods are restricted to exogenous features (independent of actions), overlooking endogenous features (dependent on actions) that may be crucial for credit assignment. We introduce the future-informed value function, which can be conditioned on both exogenous and endogenous features of future states, actions, and rewards. Then, we express the policy gradient using this value function, yielding the future-informed policy gradient. Building on that, we propose the future-informed actor-critic algorithm. Empirical results show that this algorithm leveraging endogenous features of the future outperforms standard actor-critic algorithms in environments with noisy and sparse rewards.
BibTeX
@misc{mokeddem2026future,
  author = {Samy Mokeddem and Gaspard Lambrechts and Adrien Bolland and Damien Ernst},
  title = {{Future-Informed Actor-Critic}},
  year = {2026},
  eprint = {2611.09417},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.09417},
}
Figure