Inferring Transition Dynamics from Value Functions

Jacob AdamczykarXiv 2025 · DOI · arXiv

Abstract

In reinforcement learning (RL), value functions are primarily used to rank actions. This paper reveals that a converged value function implicitly contains a complete model of the environment's transition dynamics. By "inverting" the Bellman equation, we propose a theoretical framework to extract a predictive world model from pre-existing value functions. This allows for significant knowledge reuse, potentially enabling agents to adapt to new tasks without additional environment interactions. We establish conditions for identifiability in both continuous and discrete spaces and provide empirical validation in a tabular gridworld.

Introduction

The field of reinforcement learning is often divided into model-free and model-based approaches. While model-based methods offer superior sample efficiency by learning a "world model," they are notoriously difficult to train from scratch. Conversely, model-free methods produce robust value functions that are frequently discarded once a specific task is completed.

This research addresses the opportunity for knowledge reuse by demonstrating that the physical laws of an environment—its transition dynamics—are hidden within these value functions. If the underlying physics remain constant while rewards change, we can "rescue" this dynamics information to solve novel tasks.

Theoretical Framework: Bellman Inversion

The core methodology relies on rearranging the Bellman Equation. In a deterministic environment, the value of the successor state $s'$ is directly related to the current state-action pair $(s, a)$ and the reward $r(s, a)$.

The Inversion Mechanism

Under deterministic dynamics $s' = f(s, a)$, the Bellman equation can be written as: $$V^\pi(f(s, a)) = \frac{Q^\pi(s, a) - r(s, a)}{\gamma}$$ If the value function $V$ is invertible, the next state can be solved for directly: $s' = V^{-1}(\dots)$. Geometrically, this involves finding the state $s'$ whose value matches the target value calculated from the current state's $Q$-value and reward.

The geometric intuition of Bellman inversion: the next state $s'$ is identified by finding the state whose value matches the target value derived from the current state-action pair.

Identifiability and Separability

To ensure a unique solution for $s'$, the value function must satisfy specific properties:

  • Reverse Lipschitz Continuity: In continuous spaces, the value function must be "steep" enough such that small changes in state result in distinguishable changes in value.
  • $\delta$-Separability: In discrete spaces, every state must possess a distinct value.

Counter-intuitively, this method benefits from large gradients. While smooth functions are often preferred in standard machine learning to prevent overfitting, "steeper" value landscapes make it easier to pinpoint the exact successor state.

Empirical Validation

The framework was tested using a $5 \times 5$ tabular gridworld. To evaluate the robustness of the theory, 20 different Markov Decision Processes (MDPs) were generated by sampling rewards uniformly at random while keeping the grid's physical movement rules constant.

Accuracy and Error Bounds

Experiments confirmed that when the value function is computed with high precision, the accuracy of the predicted next-state is 100%. However, real-world value functions are often only $\epsilon$-accurate. The paper provides formal bounds (Theorems 1 and 2) relating this error to the model's predictive accuracy.

Accuracy of inferred dynamics as a function of value iteration tolerance. Higher value separability ($\delta_V$) maintains higher accuracy for longer.

As shown in the figure above, the model's accuracy remains high until the error in the value function (tolerance $\epsilon$) exceeds a critical threshold relative to the state separation ($\delta$). Beyond this point, the ability to uniquely identify successor states degrades.

Discussion and Future Work

This work suggests that value functions are compressed representations of environment physics. By intersecting "level sets" (states sharing the same value) from multiple tasks, agents can "triangulate" successor states even when a single value function is not fully invertible.

Limitations

  • Stochasticity: The current framework assumes deterministic dynamics. Extending this to stochastic environments would require predicting probability distributions over states.
  • Known Parameters: The method assumes the reward function and discount factor are known.
  • High-Dimensional Inversion: Inverting neural network value functions in continuous spaces (e.g., pixel-based inputs) remains a challenge, though Invertible Neural Networks (INNs) offer a promising research direction.

In conclusion, "rescuing" dynamics from value functions provides a novel bridge between model-free and model-based RL, offering a path toward more efficient and transferable agent intelligence.

Summary updated

Abstract

In reinforcement learning, the value function is typically trained to solve the Bellman equation, which connects the current value to future values. This temporal dependency hints that the value function may contain implicit information about the environment's transition dynamics. By rearranging the Bellman equation, we show that a converged value function encodes a model of the underlying dynamics of the environment. We build on this insight to propose a simple method for inferring dynamics models directly from the value function, potentially mitigating the need for explicit model learning. Furthermore, we explore the challenges of next-state identifiability, discussing conditions under which the inferred dynamics model is well-defined. Our work provides a theoretical foundation for leveraging value functions in dynamics modeling and opens a new avenue for bridging model-free and model-based reinforcement learning.
BibTeX
@misc{adamczyk2025inferring,
  author = {Jacob Adamczyk},
  title = {{Inferring Transition Dynamics from Value Functions}},
  year = {2025},
  eprint = {2501.09081},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2501.09081},
}
Figure