Recurrent Level Set Networks for Instance Segmentation

Ngan Le, Khoa Luu, Marios Savvides, Kha Gia Quach, Chi Nhan DuongIntechOpen 2019 · DOI · OpenAlex

Abstract

Level set (LS)-based segmentation is a powerful tool in medical imaging, yet it often struggles with multi-instance objects in complex real-world scenes and exhibits high sensitivity to initialization parameters. This paper presents Contextual Recurrent Level Sets (CRLS), a framework that reformulates the classic variational level set method as a hidden state evolution within a Gated Recurrent Unit (GRU). By integrating fitting forces and contour length into the GRU's gating mechanism, the curve deformation process becomes learnable and differentiable. CRLS achieves state-of-the-art performance on PASCAL VOC 2012 and MS COCO while providing a massive speedup over traditional iterative solvers.

Introduction: The Challenge of Instance Segmentation

Instance segmentation—the task of identifying and outlining individual objects at the pixel level—remains a core challenge in computer vision. Historically, researchers utilized Level Set (LS) methods, which treat a contour as the zero-crossing of a higher-dimensional function that evolves toward object boundaries based on mathematical forces.

While LS methods are mathematically elegant, they face significant hurdles in "in-the-wild" scenarios:

  1. Sensitivity: They are highly sensitive to initial settings and the number of iterations.
  2. Complexity: They struggle to separate multiple instances of the same class (e.g., people in a crowd).
  3. Speed: Traditional iterative solvers are computationally expensive and slow.

The authors address these issues by promoting classic LS methods into a trainable deep learning framework.

Methodology: Contextual Recurrent Level Sets (CRLS)

The core innovation of this work is the Recurrent Level Set (RLS). The authors observed that the iterative nature of LS contour evolution mirrors the hidden state updates in a Recurrent Neural Network (RNN).

Recurrent Level Set (RLS)

By mapping the level set function ($\phi$) to the hidden state ($h_t$) of a GRU, the evolution process becomes differentiable and learnable via backpropagation. This allows the network to learn the optimal "path" for a contour to take toward an object boundary, rather than relying on fixed mathematical forces.

A taxonomy of Recurrent Neural Network (RNN) architectures. The RLS component leverages these structures to model the iterative evolution of object contours.

The CRLS Pipeline

To handle multiple instances effectively, the authors developed a three-stage pipeline:

  1. Detection Stage: Uses a VGG-16 backbone and a Region Proposal Network (RPN) to identify bounding boxes for potential objects.
  2. Segmentation Stage: For each box, the RLS (the GRU-based level set) evolves a mask. This "one-to-many" process takes the initial detection features and iteratively refines the object boundary.
The one-to-many recurrent structure specifically used to evolve a single detection into a sequence of mask refinements.
  1. Classification Stage: A fully-connected network uses the generated mask and features to predict the object's class.

Key Insights

  • Contour Evolution as Sequence Modeling: Treating time steps of a mathematical evolution as sequence steps in an RNN is a profound shift that enables end-to-end training.
  • Efficiency through Learning: Traditional LS methods can take over 10 seconds to converge. By learning the weights, RLS performs the same task in milliseconds (0.008s).
  • Contextual Integration: By combining RLS with a detection network, the model gains the "context" needed to apply level sets to specific instances, solving the multi-instance problem.

Results

The CRLS model demonstrated superior performance in both accuracy and speed:

  • Accuracy: On PASCAL VOC 2012, CRLS achieved a 66.7% mAP (mean Average Precision), outperforming other state-of-the-art methods like MNC (63.5%) and SDS (49.7%).
  • Speed: The RLS component is remarkably fast, reducing processing time from 13.5 seconds (baseline Chan-Vese) to just 0.008 seconds.
  • Robustness: Qualitative results show the model successfully separating overlapping objects where traditional methods often merge them.

Conclusion and Future Work

CRLS successfully bridges the gap between classical variational methods and modern deep learning. While it shows strong results on standard benchmarks, challenges remain in extremely cluttered scenes with heavy occlusion, as seen in MS COCO results. Future work may extend this recurrent formulation to 3D medical imaging volumes, where traditional level sets are already a staple but suffer from similar speed and sensitivity issues.

Summary updated

Abstract

Level set (LS)-based segmentation has been widely used in medical imaging domain. It however has some difficulty when dealing with multi-instance objects in the real world. Furthermore, LS’s performance is generally quite sensitive to some initial settings and parameters such as the number of iterations. To address these issues and promote the classic LS methods to a new degree of performance in a trainable deep learning framework, we are presenting a novel approach contextual recurrent level sets (CRLS) for object instance segmentation. In the proposed networks, the curve deformation process is formed as a hidden state evolution procedure in gated recurrent units (GRUs) and updated by minimizing an energy functional composed of fitting forces and contour length.
BibTeX
@inproceedings{le2019recurrent,
  author = {Ngan Le and Khoa Luu and Marios Savvides and Kha Gia Quach and Chi Nhan Duong},
  title = {{Recurrent Level Set Networks for Instance Segmentation}},
  booktitle = {IntechOpen},
  year = {2019},
  doi = {10.5772/intechopen.84675},
}
Figure