In the quest to build intelligent agents, we often focus on the "how" of learning—designing more efficient optimization algorithms, better reward signals, or more stable gradient updates. However, a growing body of research suggests that the ultimate performance of a reinforcement learning (RL) agent is not determined by the optimization process itself, but by the representational capacity of the underlying model.
If an agent’s neural network or basis functions cannot mathematically represent the optimal policy, no amount of training data or algorithmic refinement will bridge that gap. This "representational ceiling" acts as a fundamental bottleneck across domains ranging from robotics to large language model (LLM) alignment.
The Irreducible Error Floor
When we use function approximators like neural networks to estimate value functions or policies, we introduce an inherent approximation error. In An Improved Analysis of (Variance-Reduced) Policy Gradient and Natural Policy Gradient Methods, Kaiqing Zhang and colleagues demonstrate that while modern algorithms can achieve global convergence, they only converge to a "neighborhood" of the optimal solution. The size of this neighborhood is dictated by the expressive power of the chosen architecture.
This concept is often framed as an irreducible error floor. As Carlo Alfano and Patrick Rebeschini show in A Novel Framework for Policy Mirror Descent with General Parametrization and Linear Convergence, even when an algorithm achieves linear convergence, it eventually hits a performance plateau defined by the approximation power of the model. If the network is too shallow or the feature set too sparse, the agent will inevitably stall, regardless of the optimization's mathematical elegance.
Why "Good" Representations Aren't Enough
A common intuition in RL is that providing an agent with "good" features—low-dimensional summaries of the environment—should make learning easy. However, Simon Shaolei Du and his co-authors provide a sobering reality check in Is a Good Representation Sufficient for Sample Efficient Reinforcement Learning?. They prove that unless a representation is nearly perfect, RL can remain exponentially difficult.
This sensitivity means that even minor approximation errors can transform a tractable problem into a "needle-in-a-haystack" search. The success of an agent depends on a high-fidelity match between the model's hypothesis space and the environment's true dynamics. For example, in ConstrainedSQL: Training LLMs for Text2SQL via Constrained Reinforcement Learning, Weiqin Chen and colleagues found that the primary limit on performance was the architectural capacity of the LLM to represent the ideal SQL-generation policy, rather than the RL optimization itself.
Expressivity as a Solution to Multi-Modality
In complex environments, the "best" action might not be a single point, but a choice between several equally valid paths (multi-modality). Traditional Gaussian policies often struggle here because they can only represent a single "average" action.
To overcome this, researchers are moving toward more expressive policy classes. Zhendong Wang and Mingyuan Zhou introduced Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learning, using conditional diffusion models to capture intricate, multi-modal data distributions that simpler models miss. Similarly, Sam Lobel and his team developed Q-functionals for Value-Based Continuous Control, which represent the Q-function as a combination of basis functions to allow for more flexible action selection in robotics.
The Stability Gap
The reliance on function approximation doesn't just limit performance; it can also introduce instability. In the foundational paper Generalization in Reinforcement Learning: Safely Approximating the Value Function, Justin A. Boyan and Andrew W. Moore 0001 demonstrated that combining dynamic programming with function approximators is fundamentally unstable.
This instability arises because the model may exhaust its representational resources on irrelevant parts of the state space, leading to "divergence" where the learned values lose all meaning. John N. Tsitsiklis addressed this in Stable LInear Approximations to Dynamic Programming for Stochastic Control Problems with Local Transitions, proving that the error of a converged linear approximation is strictly bounded by the best possible approximation error within the selected feature space.
Conclusion: Designing for the Ceiling
The recurring lesson across these papers is that the "brain" of the agent—its architecture—defines its potential. Whether it is Haoxing Tian proving convergence for multi-layer neural networks or researchers optimizing production bidding policies, the conclusion remains the same: optimization can only take you as far as your representation allows. To build better agents, we must focus as much on the expressivity and alignment of our model architectures as we do on the algorithms that train them.