Abstract
Visual Reinforcement Learning (RL) typically relies on pixel-to-action mappings, which are highly sensitive to visual perturbations. We introduce SegDAC, a framework that transforms raw visual observations into a variable-length set of "dynamic object tokens" using frozen, text-grounded vision models. This object-centric representation enables agents to generalize across diverse visual environments without the need for extensive data augmentation or auxiliary reconstruction losses. Evaluated on the ManiSkill3 benchmark, SegDAC demonstrates up to 88% improvement in visual generalization over prior methods while matching the sample efficiency of top-tier pixel-based RL algorithms.
Introduction
Standard visual RL agents often suffer from extreme "brittleness." An agent trained to manipulate a red cube on a gray surface may fail entirely if the background color changes or lighting conditions shift. This stems from the "pixel-to-action" paradigm, where task-relevant objects and irrelevant environmental noise are entangled within a grid of pixels. In contrast, humans perceive the world as a collection of persistent objects.
SegDAC (Segmentation-Driven Actor-Critic) bridges this gap by leveraging modern, frozen vision models to provide an object-level view of the world. By moving away from pixel-pushing toward object-centric reasoning, SegDAC achieves remarkable robustness to visual chaos while maintaining high sample efficiency.
Methodology
The SegDAC framework employs a three-stage pipeline to process visual observations into actionable tokens at every timestep.
1. Text-Grounded Segmentation
Rather than learning segmentation from scratch, SegDAC utilizes YOLO-World, an open-vocabulary detector, to identify objects based on text prompts (e.g., "robot," "cube," "background"). These detections serve as prompts for EfficientViT-SAM, which generates high-quality instance masks.
2. Dynamic Token Construction
Features are extracted from a frozen vision encoder (such as DINOv2). By pooling features specifically from masked regions, the system creates a unique "token" for each object. To ensure spatial awareness, Segment Positional Encoding (SPE) is applied, injecting the 2D coordinates of the object's bounding box into the token.
3. Transformer Actor-Critic
These tokens are processed by a Transformer-based architecture capable of handling a variable-length input set. This flexibility allows the agent to naturally adapt as objects enter or leave the frame or become occluded. The Transformer's attention mechanism handles these changes without requiring fixed "slots" or rigid input dimensions.
Experiments and Results
The authors evaluated SegDAC on eight manipulation tasks within the ManiSkill3 environment. To rigorously test robustness, a Visual Generalization Benchmark was established, featuring 12 perturbation types across three difficulty levels (Easy, Medium, and Hard).
Key Findings
- Superior Generalization: In "Hard" benchmark scenarios—involving aggressive out-of-distribution changes like semantic confusion between objects and tables—SegDAC outperformed existing visual generalization methods by 88%. In "Medium" settings, it showed a 66% improvement.
- Sample Efficiency: Unlike many robust methods that suffer a "generalization tax" (slower learning), SegDAC matches the sample efficiency of DrQ-v2, a leading pixel-based RL method.
- Computational Efficiency: SegDAC is significantly faster than previous segmentation-based approaches, achieving roughly 1.8x the speed of SAM-G and 5.8x the speed of FTD. This makes it practical for standard research timelines, with training completed in approximately 24 hours on a single A100 GPU.
Discussion and Conclusion
SegDAC demonstrates that reasoning over objects, rather than pixels, is a powerful inductive bias for visual RL. By utilizing frozen, pre-trained vision models, the framework avoids the pitfalls of learning representations from scratch or relying on reconstruction losses.
Future work may involve extending the positional encodings to 3D, incorporating temporal memory for handling long-term occlusions, and conditioning policies on natural language instructions. While currently tested in simulation, the aggressive visual perturbations suggest strong potential for successful sim-to-real transfer.
Summary updated