Generator evaluator-selector net: a modular approach for panoptic segmentation

Sagi Eppel, Alán Aspuru-Guzik

Introduction

Panoptic segmentation is a comprehensive scene understanding task that requires identifying individual object instances ("things") while simultaneously labeling background regions ("stuff"). Traditional models, such as Mask R-CNN, are typically class-specific and struggle with unfamiliar categories. This paper introduces the Generator-Evaluator-Selector (GES) framework, a modular approach that treats segmentation as a trial-and-error process. By decoupling the generation of candidate segments from their evaluation, the system achieves robust, class-agnostic segmentation, enabling it to decompose novel objects into their constituent parts without category-specific training.

Methodology

The GES framework is inspired by evolutionary trial-and-error and consists of four independent neural components:

  1. Generator (Pointer Net): This network takes an image and a specific coordinate ("pointer point") to predict a segment containing that point. By sampling various points, it generates a diverse set of candidate segments.
  2. Evaluator: Acting as a judge, this network predicts the quality (Intersection over Union, or IoU) of each candidate segment, filtering out low-quality proposals.
  3. Refiner: This component polishes the boundaries of selected segments to improve spatial accuracy.
  4. Classifier: An independent network that assigns semantic category labels to the final segments.
The GES framework applied to panoptic and parts segmentation tasks, illustrating the modular 'guess and check' pipeline.

The process is iterative: the system identifies segments, integrates them into a master map, and repeats the cycle on remaining unsegmented regions until the entire scene is processed.

Experimental Setup

The researchers evaluated the GES framework on three major benchmarks: COCO Panoptic for general scene segmentation, and ADE20K and Pascal Parts for object-part decomposition. To test class-agnostic performance, the authors intentionally withheld 20% of object classes during training for the parts segmentation task, forcing the model to generalize to unfamiliar categories.

Key Results and Discussion

Performance and Generalization

On the COCO benchmark, the GES system achieved a Panoptic Quality (PQ) score of 33.7. While slightly lower than specialized state-of-the-art models, it remains competitive with baseline Mask R-CNN implementations. Notably, the system excelled at class-agnostic part discovery. On ADE20K, it achieved an IoU of 59 for unfamiliar classes—slightly outperforming its performance on familiar classes (58). This demonstrates that the model effectively learns geometric concepts of "parts" rather than just memorizing labels.

Demonstration of the iterative segmentation process for complex objects, showing how the evaluator selects high-quality part masks.

The Importance of Evaluation

Ablation studies highlighted the Evaluator as a critical component; its removal led to a 9-point drop in PQ score. This supports the core philosophy that ranking solutions is fundamentally easier and more robust than generating a perfect solution in a single pass. The authors liken this to guessing a 1000-letter string: while guessing the whole string at once is nearly impossible ($9^{1000}$ combinations), evaluating and selecting each letter independently reduces the search space to $9 \times 1000$ tries. By segmenting one piece at a time, the search space is reduced exponentially compared to whole-image prediction.

Conclusion

The GES framework demonstrates that a modular, iterative approach to image segmentation can achieve high-quality results and exceptional generalization to unseen object classes. While currently limited by the accuracy of the independent Classifier net—particularly for "stuff" categories—future work could integrate more advanced classification techniques or explore active learning applications where the evaluator identifies high-uncertainty regions for refinement.

Summary updated

Abstract

In machine learning and other fields, suggesting a good solution to a problem is usually a harder task than evaluating the quality of such a solution. This asymmetry is the basis for a large number of selection oriented methods that use a generator system to guess a set of solutions and an evaluator system to rank and select the best solutions. This work examines the use of this approach to the problem of panoptic image segmentation and class agnostic parts segmentation. The generator/evaluator approach for this case consists of two independent convolutional neural nets: a generator net that suggests variety segments corresponding to objects, stuff and parts regions in the image, and an evaluator net that chooses the best segments to be merged into the segmentation map. The result is a trial and error evolutionary approach in which a generator that guesses segments with low average accuracy, but with wide variability, can still produce good results when coupled with an accurate evaluator. The generator consists of a Pointer net that receives an image and a point in the image, and predicts the region of the segment containing the point. Generating and evaluating each segment separately is essential in this case since it demands exponentially fewer guesses compared to a system that guesses and evaluates the full segmentation map in each try. The classification of the selected segments is done by an independent region-specific classification net. This allows the segmentation to be class agnostic and hence, capable of segmenting unfamiliar categories that were not part of the training set. The method was examined on the COCO Panoptic segmentation benchmark and gave results comparable to those of the basic semantic segmentation and Mask-RCNN methods. In addition, the system was used for the task of splitting objects of unseen classes (that did not appear in the training set) into parts.
BibTeX
@misc{eppel2019generator,
  author = {Sagi Eppel and Alán Aspuru-Guzik},
  title = {{Generator evaluator-selector net: a modular approach for panoptic segmentation}},
  year = {2019},
  eprint = {1908.09108},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1908.09108},
}
Figure