Abstract
Most approaches to visual scene analysis have emphasised parallel processing of the image elements. However, one area in which the sequential nature of vision is apparent, is that of segmenting multiple, potentially similar and partially occluded objects in a scene. In this work, we revisit the recurrent formulation of this challenging problem in the context of reinforcement learning. Motivated by the limitations of the global max-matching assignment of the ground-truth segments to the recurrent states, we develop an actor-critic approach in which the actor recurrently predicts one instance mask at a time and utilises the gradient from a concurrently trained critic network. We formulate the state, action, and the reward such as to let the critic model long-term effects of the current prediction and incorporate this information into the gradient signal. Furthermore, to enable effective exploration in the inherently high-dimensional action space of instance masks, we learn a compact representation using a conditional variational auto-encoder. We show that our actor-critic model consistently provides accuracy benefits over the recurrent baseline on standard instance segmentation benchmarks.
Introduction: The Sequential Nature of Vision
Most modern instance segmentation methods, such as Mask R-CNN, operate in parallel by proposing numerous bounding boxes and predicting masks simultaneously. While effective, this approach often struggles with heavy occlusion or scenes containing many similar objects. Researchers from TU Darmstadt have revisited this problem by framing instance segmentation as a sequential decision-making process using Reinforcement Learning (RL).
The Problem: Ordering and Horizons
Recurrent models for segmentation face two primary hurdles. First is the Ordering Problem: determining the optimal sequence for segmenting objects. Traditional methods rely on the Kuhn-Munkres algorithm for global max-matching, which fails to account for how early errors propagate through the sequence. Second is the Horizon Problem: training recurrent models on high-resolution images using standard backpropagation through time (BPTT) is memory-prohibitive, often forcing researchers to truncate training and ignore long-term dependencies.
Methodology: The Actor-Critic Architecture
To address these challenges, the authors propose an Actor-Critic (AC) framework that treats segmentation as a series of actions taken by an agent.
- The Actor: A recurrent network (using an LSTM) that predicts one instance mask at each timestep. To manage the "prohibitively high-dimensional" action space of image pixels, the authors use a Conditional Variational Auto-Encoder (cVAE). This compresses a full-resolution mask into a compact latent vector (the "action"), which the decoder then expands back into a mask.
- The Critic: Instead of evaluating immediate error, the Critic network estimates the future reward. It provides a gradient signal to the Actor that accounts for the long-term impact of a prediction—for instance, prioritizing a foreground object to simplify the segmentation of occluded objects later.
- State and Reward: The model's state includes the original image and a "State Pyramid"—an aggregated mask of everything already segmented. Rewards are based on the improvement in the "state potential" (measured by Dice score or IoU) after each new prediction.
Experimental Evaluation
The model was evaluated on two distinct benchmarks:
- CVPPP (Leaf Segmentation): A dataset characterized by significant leaf overlap, testing the model's ability to count and segment similar, occluded objects.
- KITTI (Traffic Scenes): A real-world dataset of cars in street scenes, testing scalability to higher resolutions and varying object scales.
Key Insights
- Natural Curriculum Learning: Without explicit instruction, the model learned to segment "easy" (large, unoccluded) objects first. This clears the visual field, making it easier to tackle "hard" (small, distant, or occluded) objects later.
- Breaking the Memory Wall: Because the Critic models future rewards, the system avoids the need to unroll the entire sequence for gradient backpropagation. This allows for training on longer sequences and higher resolutions than traditional recurrent methods.
Results and Conclusion
On the CVPPP dataset, the Actor-Critic model achieved a Symmetric Best Dice (SBD) of 80.5, outperforming recurrent baselines and maintaining higher quality for the final objects in a sequence. On KITTI, the model demonstrated it could handle higher resolutions and successfully segmented cars from near to far without requiring bounding box proposals.
While the authors note that a gap remains compared to the raw pixel-level accuracy of the best proposal-based pipelines, this work provides a strong proof-of-concept. It demonstrates that Reinforcement Learning, combined with action space compression, is a viable path for solving complex, high-resolution computer vision tasks one step at a time.
Summary updated