Abstract
In reinforcement learning (RL), value functions are primarily used to rank actions. This paper reveals that a converged value function implicitly contains a complete model of the environment's transition dynamics. By "inverting" the Bellman equation, we propose a theoretical framework to extract a predictive world model from pre-existing value functions. This allows for significant knowledge reuse, potentially enabling agents to adapt to new tasks without additional environment interactions. We establish conditions for identifiability in both continuous and discrete spaces and provide empirical validation in a tabular gridworld.
Introduction
The field of reinforcement learning is often divided into model-free and model-based approaches. While model-based methods offer superior sample efficiency by learning a "world model," they are notoriously difficult to train from scratch. Conversely, model-free methods produce robust value functions that are frequently discarded once a specific task is completed.
This research addresses the opportunity for knowledge reuse by demonstrating that the physical laws of an environment—its transition dynamics—are hidden within these value functions. If the underlying physics remain constant while rewards change, we can "rescue" this dynamics information to solve novel tasks.
Theoretical Framework: Bellman Inversion
The core methodology relies on rearranging the Bellman Equation. In a deterministic environment, the value of the successor state $s'$ is directly related to the current state-action pair $(s, a)$ and the reward $r(s, a)$.
The Inversion Mechanism
Under deterministic dynamics $s' = f(s, a)$, the Bellman equation can be written as: $$V^\pi(f(s, a)) = \frac{Q^\pi(s, a) - r(s, a)}{\gamma}$$ If the value function $V$ is invertible, the next state can be solved for directly: $s' = V^{-1}(\dots)$. Geometrically, this involves finding the state $s'$ whose value matches the target value calculated from the current state's $Q$-value and reward.
Identifiability and Separability
To ensure a unique solution for $s'$, the value function must satisfy specific properties:
- Reverse Lipschitz Continuity: In continuous spaces, the value function must be "steep" enough such that small changes in state result in distinguishable changes in value.
- $\delta$-Separability: In discrete spaces, every state must possess a distinct value.
Counter-intuitively, this method benefits from large gradients. While smooth functions are often preferred in standard machine learning to prevent overfitting, "steeper" value landscapes make it easier to pinpoint the exact successor state.
Empirical Validation
The framework was tested using a $5 \times 5$ tabular gridworld. To evaluate the robustness of the theory, 20 different Markov Decision Processes (MDPs) were generated by sampling rewards uniformly at random while keeping the grid's physical movement rules constant.
Accuracy and Error Bounds
Experiments confirmed that when the value function is computed with high precision, the accuracy of the predicted next-state is 100%. However, real-world value functions are often only $\epsilon$-accurate. The paper provides formal bounds (Theorems 1 and 2) relating this error to the model's predictive accuracy.
As shown in the figure above, the model's accuracy remains high until the error in the value function (tolerance $\epsilon$) exceeds a critical threshold relative to the state separation ($\delta$). Beyond this point, the ability to uniquely identify successor states degrades.
Discussion and Future Work
This work suggests that value functions are compressed representations of environment physics. By intersecting "level sets" (states sharing the same value) from multiple tasks, agents can "triangulate" successor states even when a single value function is not fully invertible.
Limitations
- Stochasticity: The current framework assumes deterministic dynamics. Extending this to stochastic environments would require predicting probability distributions over states.
- Known Parameters: The method assumes the reward function and discount factor are known.
- High-Dimensional Inversion: Inverting neural network value functions in continuous spaces (e.g., pixel-based inputs) remains a challenge, though Invertible Neural Networks (INNs) offer a promising research direction.
In conclusion, "rescuing" dynamics from value functions provides a novel bridge between model-free and model-based RL, offering a path toward more efficient and transferable agent intelligence.
Summary updated