TL;DR
This paper demonstrates that using pre-trained "mid-level" visual features (like depth estimation or edge detection) significantly improves the sample efficiency and generalization of reinforcement learning agents for robotic tasks, compared to learning from raw pixels. They also introduce a method to select an optimal subset of these features for any given task.
What Problem Does This Paper Address?
Deep reinforcement learning (RL) agents have shown impressive capabilities in learning complex tasks directly from raw pixel inputs (e.g., playing video games or controlling robots). However, this "pixel-to-torque" approach suffers from two major drawbacks: it requires an enormous amount of training data (low sample efficiency) and the resulting policies often fail to generalize to new, unseen environments with even minor visual differences. This raises a fundamental question: are prior visual assumptions about the world (like its 3D structure or object properties) necessary for efficient and robust learning, or can RL learn everything from scratch given enough data?
Methods
The core idea is to augment the standard RL observation space, which typically consists of raw images, with features extracted from pre-trained "mid-level" vision models. These models are trained on specific, generic vision tasks (e.g., estimating surface normals, detecting edges, segmenting scenes) using a large dataset of static images. The authors then freeze the weights of these pre-trained models and use their internal representations as the input state for the RL agent. This means the RL agent learns a policy based on these processed features rather than raw pixels.
To address the question of which features are most beneficial, they propose a "Max-Coverage Feature Selector." This method uses a measure of distance between perceptual tasks to curate a compact subset of features that minimizes the worst-case distance to the ideal feature for a given task. This aims to find a small, general-purpose set of features that can be effective across a range of downstream tasks.
Datasets
The paper uses the Gibson environment [86], a dataset of 3D indoor scenes designed to be perceptually similar to the real world. They train and test agents in disjoint sets of buildings within this environment to evaluate generalization. The mid-level vision features themselves are derived from networks trained on a dataset of 4 million static images of indoor scenes [90].
Key Insights and Ideas
- Visual Priors are Crucial for Efficiency and Generalization: The study strongly suggests that incorporating prior knowledge about the visual world, even in the form of generic mid-level features, is not just helpful but essential for efficient learning and robust generalization in RL, especially in data-limited scenarios.
- "Tabula Rasa" Learning is Brittle: Agents trained from scratch on raw pixels tend to overfit to the training environment, performing poorly in unseen test environments, sometimes even worse than a "blind" agent that receives no visual input.
- Task-Specific Features Matter: There isn't a single "best" visual feature that works for all tasks. For instance, depth estimation features are good for exploration, while object classification features are better for navigation to a specific target. This "rank reversal" phenomenon highlights the need for a set of features.
- A Compact Feature Set is Sufficient: A carefully selected small subset of mid-level features (e.g., 4 features) can achieve performance comparable to or even exceeding the best single task-specific feature, while remaining agnostic to the specific downstream task. This offers a practical way to leverage mid-level vision without needing to train a massive ensemble of features.
Key Results
- Sample Efficiency: Feature-based agents learned significantly faster than scratch-based agents, achieving higher rewards with fewer training frames across all tested tasks (Visual-Target Navigation, Visual Exploration, Local Planning). Figure 7 shows that feature-based agents reach peak performance much earlier.
- Generalization: Feature-based agents generalized much better to unseen test environments. Figure 9 clearly illustrates that while scratch agents often perform poorly in test environments, feature-based agents maintain a much smaller gap between training and test performance.
- Rank Reversal: The study confirmed rank reversal, showing that features optimal for one task (e.g., object classification for navigation) were suboptimal for another (e.g., exploration). This was statistically significant.
- Max-Coverage Feature Set: A set of just 4 selected features was able to nearly match or exceed the performance of the best task-specific feature, demonstrating the effectiveness of their selection method.
Limitations and Future Work
- Model Dependence: The study uses neural networks as the function class for both the mid-level vision models and the RL policies. The findings might differ if other models (e.g., nearest neighbors) were used.
- Fixed Feature Set: The set of mid-level features used was fixed. Future work could explore how to continually update these visual estimators and incrementally expand the dictionary of features.
- Locomotive Tasks: The active tasks were primarily locomotion-based. The utility of mid-level vision on other tasks, such as manipulation, remains an open question.
- Limited Representation Set: The authors froze the feature weights, which limits expressiveness. Relaxing this constraint could potentially improve performance further.
- Lack of Guarantees: The approach is empirical, and guarantees for agent success or perceptual distance are not provided. Predicting agent performance in a new environment would be important for real-world deployment.
- Lifelong Learning: The current mid-level feature set is static. Investigating how to incrementally update and expand this set for lifelong learning is a key future direction.
Summary updated