In traditional Reinforcement Learning (RL), the "state" of the world is often handed to the agent on a silver platter. To train a robot to navigate a room, a human engineer might define features like "distance to the nearest wall" or "angle to the goal." While effective for simple tasks, this manual feature engineering creates a significant bottleneck. As Yanping Huang notes, relying on hand-coded basis functions is inefficient, error-prone, and prevents agents from adapting to unforeseen environments.
The research direction emerging across these papers is the move toward automated representation learning: systems that autonomously discover the underlying structure, utility, and features of an environment directly from raw data.
Inferring Utility from Movement
One of the most intuitive ways to bypass manual labeling is to treat an agent's behavior as a map of its desires. In Learning from lions: inferring the utility of agents from their trajectories, Adam D. Cobb, Andrew Markham, and S Roberts demonstrate that we don't need to pre-define landmarks to understand what a lion (or a robot) wants.
By treating movement acceleration as the gradient of a potential field, they use Gaussian Processes to identify "sources" and "sinks"—attractors and repellers in the environment. This allows the discovery of novel drivers of behavior without a human expert having to label every water hole or predator territory in advance.
Discovering Geometric and Algebraic Structure
If the environment is a high-dimensional maze of pixels, how does an agent find a manageable "shape" for its learning? Researchers are increasingly looking at the mathematical properties of the state space itself to find these shortcuts.
- Low-Rank Approximations: Devavrat Shah and colleagues in Sample Efficient Reinforcement Learning via Low-Rank Matrix Estimation show that optimal value functions ($Q$-functions) often possess a hidden low-rank structure. By exploiting this, agents can learn optimal policies with exponentially fewer data points, effectively "compressing" the complexity of the world.
- Spectral Graph Theory: Sridhar Mahadevan introduced Representation Policy Iteration (RPI), which uses the geometry of the state-space graph to generate basis functions. Instead of using generic polynomials, the agent learns functions that respect the actual topology of its environment, such as recognizing that two points separated by a wall are "far apart" even if they are physically close in 3D space.
Composable Behaviors and the Successor Feature Keyboard
A major challenge in general AI is scaling: how do we move from learning one task to mastering many? Wilka Carvalho and Andre Saraiva propose the Successor Features Keyboard (SFK).
The intuition here is to treat behaviors like musical notes. Instead of hand-designing how tasks are encoded, the agent uses a categorical learning algorithm to discover its own internal "basis behaviors." Much like a keyboard can play an infinite variety of songs from a finite set of keys, the agent can compose these learned features to solve new, long-horizon tasks in 3D environments without needing humans to define the sub-goals.
Automating the Expert
Even when we have human experts available, the bottleneck remains: how do we translate their "gut feeling" into math?
Sydney M. Katz addresses this in the context of Urban Air Mobility. By using preference-based learning—simply asking experts to choose which of two simulated flight paths looks "safer"—the system builds a probabilistic model of expert intent. This bypasses the need for the expert to manually define the mathematical trade-offs between speed, safety, and fuel efficiency.
More recently, researchers have begun using Large Language Models (LLMs) to bridge this gap. Yun Qu introduced LESR, a framework where an LLM automatically writes Python code to engineer physically meaningful state features and rewards. This essentially automates the role of the human engineer, allowing the RL agent to learn from "smoother" mathematical representations than raw pixels provide.
Why This Works
The shift toward automated representation learning works because it aligns the agent's internal "map" with the actual dynamics of the task, rather than a human's potentially biased or incomplete interpretation of those dynamics. Whether through spectral graph theory, low-rank estimation, or successor features, these methods allow agents to "de-alias" complex environments—distinguishing between states that look similar but have different consequences—without requiring a human to anticipate every possible scenario.