Abstract
Most approaches to visual scene analysis have emphasised parallel processing of the image elements. However, one area in which the sequential nature of vision is apparent, is that of segmenting multiple, potentially similar and partially occluded objects in a scene. In this work, we revisit the recurrent formulation of this challenging problem in the context of reinforcement learning. Motivated by the limitations of the global max-matching assignment of the ground-truth segments to the recurrent states, we develop an actor-critic approach in which the actor recurrently predicts one instance mask at a time and utilises the gradient from a concurrently trained critic network. We formulate the state, action, and the reward such as to let the critic model long-term effects of the current prediction and in-corporate this information into the gradient signal. Furthermore, to enable effective exploration in the inherently high-dimensional action space of instance masks, we learn a compact representation using a conditional variational auto-encoder. We show that our actor-critic model consistently provides accuracy benefits over the recurrent baseline on standard instance segmentation benchmarks.
Introduction
Most modern instance segmentation models, such as Mask R-CNN, process images in parallel, detecting all objects at once. However, human vision is inherently sequential; we often focus on prominent objects first before resolving cluttered or occluded ones. This paper reframes instance segmentation as a sequential decision-making process using an Actor-Critic reinforcement learning framework.
Existing recurrent (sequential) models face three major hurdles:
- The Assignment Problem: During training, determining the order in which the model should predict ground-truth objects is difficult. Standard methods are often sensitive to initialization.
- High-Dimensional Actions: Predicting a full-resolution image mask is an "action" with tens of thousands of dimensions, which is computationally overwhelming for standard reinforcement learning (RL).
- Lack of Foresight: Early predictions in a sequence affect everything that follows. Standard training often ignores these long-term dependencies because backpropagating through long sequences is memory-intensive.
Methodology
The authors propose an Actor-Critic (AC) framework to solve these issues by combining recurrent neural networks with generative modeling.
The Actor and Critic
- The Actor: A recurrent neural network (LSTM) that predicts one instance mask at each timestep. To handle the high-dimensional output, it uses a Conditional Variational Auto-Encoder (cVAE). The Actor predicts a small "latent code" which a pre-trained decoder then expands into a full mask. A State Pyramid feeds multi-scale image features directly into the process to preserve fine details.
- The Critic: A network that evaluates the Actor's choices by estimating the Q-value—the expected total reward (like Dice score) from the current step until the end of the sequence. This allows the Actor to receive a gradient signal that accounts for future success, not just immediate accuracy.
Experimental Results
The model was evaluated on two distinct benchmarks: CVPPP (A1 benchmark), focusing on leaf segmentation, and KITTI, a dataset for autonomous driving.
Key Insights and Performance
The model naturally learns a "curriculum," segmenting large, unoccluded objects first to help find smaller, occluded objects later. On the CVPPP dataset, the model achieved a Symmetric Best Dice (SBD) of 80.5, outperforming the recurrent baseline (79.4).
A significant finding is the Later-Step Advantage. While standard recurrent models lose accuracy as the sequence progresses, the Actor-Critic model maintains higher accuracy at later timesteps because the Critic has "taught" it to make early moves that leave room for later success. This is particularly evident in the 15th-20th objects predicted in the CVPPP dataset.
Limitations and Future Work
While the model bridges the gap between pixels and efficient RL, it sometimes struggles with extremely fine details due to the compressed latent space. Future work could involve integrating more powerful backbones or more sophisticated reward functions to further close the gap with parallel state-of-the-art models.
Summary updated