SegDAC: Segmentation-Driven Actor-Critic for Visual Reinforcement Learning

Alexandre Brown, Glen Berseth

Abstract

Visual Reinforcement Learning (RL) typically relies on pixel-to-action mappings, which are highly sensitive to visual perturbations. We introduce SegDAC, a framework that transforms raw visual observations into a variable-length set of "dynamic object tokens" using frozen, text-grounded vision models. This object-centric representation enables agents to generalize across diverse visual environments without the need for extensive data augmentation or auxiliary reconstruction losses. Evaluated on the ManiSkill3 benchmark, SegDAC demonstrates up to 88% improvement in visual generalization over prior methods while matching the sample efficiency of top-tier pixel-based RL algorithms.

Introduction

Standard visual RL agents often suffer from extreme "brittleness." An agent trained to manipulate a red cube on a gray surface may fail entirely if the background color changes or lighting conditions shift. This stems from the "pixel-to-action" paradigm, where task-relevant objects and irrelevant environmental noise are entangled within a grid of pixels. In contrast, humans perceive the world as a collection of persistent objects.

SegDAC (Segmentation-Driven Actor-Critic) bridges this gap by leveraging modern, frozen vision models to provide an object-level view of the world. By moving away from pixel-pushing toward object-centric reasoning, SegDAC achieves remarkable robustness to visual chaos while maintaining high sample efficiency.

Methodology

The SegDAC framework employs a three-stage pipeline to process visual observations into actionable tokens at every timestep.

The high-level architecture of SegDAC, showing the flow from raw observations to Q-values via object-centric segmentation and dynamic token construction.

1. Text-Grounded Segmentation

Rather than learning segmentation from scratch, SegDAC utilizes YOLO-World, an open-vocabulary detector, to identify objects based on text prompts (e.g., "robot," "cube," "background"). These detections serve as prompts for EfficientViT-SAM, which generates high-quality instance masks.

The detailed object-centric segmentation pipeline, highlighting the use of open-vocabulary detection and morphological refinement.

2. Dynamic Token Construction

Features are extracted from a frozen vision encoder (such as DINOv2). By pooling features specifically from masked regions, the system creates a unique "token" for each object. To ensure spatial awareness, Segment Positional Encoding (SPE) is applied, injecting the 2D coordinates of the object's bounding box into the token.

3. Transformer Actor-Critic

These tokens are processed by a Transformer-based architecture capable of handling a variable-length input set. This flexibility allows the agent to naturally adapt as objects enter or leave the frame or become occluded. The Transformer's attention mechanism handles these changes without requiring fixed "slots" or rigid input dimensions.

Experiments and Results

The authors evaluated SegDAC on eight manipulation tasks within the ManiSkill3 environment. To rigorously test robustness, a Visual Generalization Benchmark was established, featuring 12 perturbation types across three difficulty levels (Easy, Medium, and Hard).

Key Findings

  • Superior Generalization: In "Hard" benchmark scenarios—involving aggressive out-of-distribution changes like semantic confusion between objects and tables—SegDAC outperformed existing visual generalization methods by 88%. In "Medium" settings, it showed a 66% improvement.
  • Sample Efficiency: Unlike many robust methods that suffer a "generalization tax" (slower learning), SegDAC matches the sample efficiency of DrQ-v2, a leading pixel-based RL method.
  • Computational Efficiency: SegDAC is significantly faster than previous segmentation-based approaches, achieving roughly 1.8x the speed of SAM-G and 5.8x the speed of FTD. This makes it practical for standard research timelines, with training completed in approximately 24 hours on a single A100 GPU.

Discussion and Conclusion

SegDAC demonstrates that reasoning over objects, rather than pixels, is a powerful inductive bias for visual RL. By utilizing frozen, pre-trained vision models, the framework avoids the pitfalls of learning representations from scratch or relying on reconstruction losses.

Future work may involve extending the positional encodings to 3D, incorporating temporal memory for handling long-term occlusions, and conditioning policies on natural language instructions. While currently tested in simulation, the aggressive visual perturbations suggest strong potential for successful sim-to-real transfer.

Summary updated

Abstract

Visual reinforcement learning policies trained on pixel observations often struggle to generalize when visual conditions change at test time. Object-centric representations are a promising alternative, but most approaches use fixed-size slot representations, require image reconstruction, or need auxiliary losses to learn object decompositions. As a result, it remains unclear how to learn RL policies directly from object-level inputs without these constraints. We propose SegDAC, a Segmentation-Driven Actor-Critic that operates on a variable-length set of object token embeddings. At each timestep, text-grounded segmentation produces object masks from which spatially aware token embeddings are extracted. A transformer-based actor-critic processes these dynamic tokens, using segment positional encoding to preserve spatial information across objects. We ablate these design choices and show that both segment positional encoding and variable-length processing are individually necessary for strong performance. We evaluate SegDAC on 8 ManiSkill3 manipulation tasks under 12 visual perturbation types across 3 difficulty levels. SegDAC improves over prior visual generalization methods by 15% on easy, 66% on medium, and 88% on the hardest settings. SegDAC matches the sample efficiency of the state-of-the-art visual RL methods while achieving improved generalization under visual changes. Project Page: https://segdac.github.io/
BibTeX
@misc{brown2025segdac,
  author = {Alexandre Brown and Glen Berseth},
  title = {{SegDAC: Segmentation-Driven Actor-Critic for Visual Reinforcement Learning}},
  year = {2025},
  eprint = {2508.09325},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2508.09325},
}
Figure