For an autonomous agent to make optimal decisions, plan ahead, or cooperate with others, it must have a working representation of its environment. Historically, this representation was mediated by human engineers who designed specific, hand-crafted features. In board games, this meant defining fixed-size local patterns; in spoken dialogue systems, it meant hand-coding belief state vectors; and in robotic control, it meant manually specifying physical factors or spatial heuristics.
This reliance on human intervention creates a severe feature engineering bottleneck. Hand-crafted features require immense domain expertise, fail to capture latent or variable-sized structures, and prevent agents from operating directly in high-dimensional real-world domains like raw pixels or text logs. When the environment changes or an agent encounters a novel scenario, these rigid human-defined heuristics break down.
To achieve true autonomy, the field has shifted toward automated feature discovery and representation learning. Instead of relying on human-provided priors, modern frameworks enable agents to self-discover the state spaces, skill hierarchies, and reward structures necessary to master complex tasks.
Discovering State and Value Structures Autonomously
When dealing with high-dimensional inputs like raw pixel data, manually designing features becomes impractical. A core focus of representation learning is compressing complex environmental histories into compact, mathematically tractable structures without losing critical information.
One prominent approach is to learn predictive representations that preserve only the information necessary to forecast future rewards and observations. For instance, Boots et al. [Predictive State Temporal Difference Learning] introduced a method to automatically compress high-dimensional, noisy history features into a compact "predictive state." This eliminates the need for manual system identification while maintaining accurate value function estimates in partially observable environments.
Alternatively, agents can exploit the underlying mathematical structure of the value function itself. Rather than requiring human experts to pre-define low-dimensional features, Shah [Sample Efficient Reinforcement Learning via Low-Rank Matrix Estimation] demonstrated that the optimal $Q$-function often possesses a hidden low-rank structure. By estimating this matrix directly from data, agents can achieve an exponential reduction in sample complexity without any prior domain knowledge.
This shift is equally vital for automated planning. Traditional width-based lookahead search algorithms rely heavily on handcrafted features to define state novelty and prune search trees. To overcome this limitation, Junyent et al. [Deep Policies for Width-Based Planning in Pixel Domains] and O'Toole et al. [Width-based Lookaheads with Learnt Base Policies and Heuristics Over the Atari-2600 Benchmark] integrated deep neural networks with width-based planners. By learning representations and base policies directly from raw pixel domains, these hybrid systems eliminate the need for manual novelty metrics, allowing planners to generalize across diverse, high-branching environments.
Unsupervised Skill and Option Discovery
The feature engineering bottleneck also plagues hierarchical reinforcement learning (HRL). For an agent to break a long-horizon task down into reusable subtasks (or "options"), it traditionally needs a human-defined feature space to identify meaningful goals or milestones.
To bypass this, researchers leverage spectral graph theory and representation learning to discover behaviors completely unsupervised. Rosenbaum et al. [Eigenoption Discovery through the Deep Successor Representation] connected deep reinforcement learning with the Successor Representation to automatically discover "eigenoptions"—hierarchical subtasks that optimize exploration—directly from raw pixel data in stochastic environments.
However, purely unsupervised agents often risk learning trivial behaviors (such as static poses) unless guided by human constraints. To prevent this without reverting to manual feature selection, Park et al. [Lipschitz-constrained Unsupervised Skill Discovery] introduced a Lipschitz continuity constraint that forces agents to discover dynamic, far-reaching behaviors. Similarly, Luketina et al. [Learning Skills Diverse in Value-Relevant Features] developed a semi-supervised framework that uses a minimal set of source tasks to identify which state features actually drive performance. This allows the agent to compress the exploration space and discover skills that are both diverse and highly relevant for downstream tasks, entirely free from human-engineered priors.
Learning Rewards and Preferences from Raw Data
A parallel challenge exists in defining what the agent should do. In inverse reinforcement learning (IRL) and preference-based learning, reward functions have traditionally been modeled as linear combinations of pre-defined, hand-coded features.
When a robot must adapt its movements to satisfy user preferences or environmental constraints, relying on hand-crafted heuristics limits its capacity to generalize to novel tasks. As highlighted by Choi et al. [Nonparametric Bayesian Inverse Reinforcement Learning for Multiple Reward Functions], there is a critical need for methods that extract reward features directly from raw sensory inputs rather than relying on manual feature engineering.
Weak supervision offers a compelling path forward. Instead of explicitly programming the physical factors an agent should care about, algorithms can learn a disentangled latent representation from simple human feedback. Eysenbach et al. [Weakly-Supervised Reinforcement Learning for Controllable Behavior] demonstrated that by using binary comparisons between images, an agent can autonomously discover salient environmental variables and ignore irrelevant background "chaff." This end-to-end feature discovery allows agents to solve complex manipulation tasks without ever receiving a manually specified reward function.
Expanding to Actions and Opponents
The necessity of moving beyond manual feature engineering extends across the entire reinforcement learning spectrum, including how agents model opponents and understand their own action spaces:
- Opponent Modeling: In multi-agent settings, treating competitors as static elements or relying on hand-crafted behavioral features (such as calculating the frequency of specific moves) limits adaptability. Daumé III et al. [Opponent Modeling in Deep Reinforcement Learning] introduced frameworks that jointly learn a policy and a continuous representation of opponent behavior directly from raw action sequences, allowing agents to dynamically adapt to novel strategies.
- Action Representations: When dealing with large or continuous action spaces, traditional methods often rely on pre-existing domain knowledge to map relationships between actions. Chandak et al. [Learning Action Representations for Reinforcement Learning] addressed this dependency by shifting toward frameworks that autonomously learn continuous action embeddings, removing the reliance on hand-crafted action features.
By replacing human-engineered templates with end-to-end representation learning, modern reinforcement learning and planning systems are overcoming their historical brittleness. Whether compressing state histories, discovering hierarchical skills, or inferring human preferences, the transition to automated feature discovery is proving essential for achieving truly scalable and generalized autonomy.