Generator evaluator-selector net for panoptic image segmentation and splitting unfamiliar objects into parts

Sagi Eppel, Alán Aspuru‐Guzik

Abstract

This paper introduces the Generator-Evaluator-Selector (GES) framework, a modular approach to image segmentation based on the principle that evaluating a solution is often simpler than generating one. The system utilizes a generator network to propose multiple candidate segments and an independent evaluator network to rank and select the most accurate ones. This decoupled architecture allows for effective panoptic segmentation and enables class-agnostic parts segmentation of unfamiliar objects. Experimental results on the COCO Panoptic and ADE20K benchmarks demonstrate that the GES framework achieves performance comparable to established methods while showing superior generalization capabilities for segmenting parts of previously unseen object categories.

Introduction

Traditional image segmentation models often struggle with novel objects or complex scenes because they are typically class-specific and attempt to predict full segmentation maps in a single pass. The GES framework addresses these challenges by adopting a "trial and error" strategy. By decomposing the problem into independent generation and evaluation phases, the model can leverage a high-variability generator alongside a precise evaluator to produce robust segmentation maps, even for unfamiliar classes.

Methodology: The GES Framework

The GES architecture consists of four specialized, independent convolutional neural networks:

  1. Generator (Pointer Net): This network takes an image and a specific "pointer point" as input, predicting a candidate segment that contains that point. By sampling various points, the generator produces a diverse set of potential regions.
  2. Evaluator: Acting as a judge, this network evaluates each proposed segment and predicts its Intersection over Union (IoU) score. It filters out low-quality guesses, ensuring only coherent regions proceed.
  3. Refinement Net: This component polishes the boundaries of selected segments to ensure precise alignment with the underlying image features.
  4. Classifier: A region-specific network that assigns semantic labels (e.g., "sky," "person," "base") to the finalized segments.
The GES framework applied to both Panoptic and Parts segmentation tasks.

The segmentation process is iterative: the system generates segments for unassigned image regions, selects the best candidates via the evaluator, and merges them into the final map. This modularity is key to its efficiency, as evaluating individual segments is exponentially less complex than evaluating entire segmentation maps.

Class-Agnostic Parts Segmentation

A significant advantage of the GES framework is its ability to perform class-agnostic parts segmentation. Because the generator and evaluator focus on regional coherence rather than specific class labels, the system can identify constituent parts of objects it has never encountered during training. For instance, it can recognize a "lid" or "base" as distinct parts of a novel object based on learned geometric and textural cues.

Detailed parts segmentation workflow illustrating the generation and evaluation of candidate segments for a complex object.

Results and Discussion

The framework was evaluated on the COCO Panoptic segmentation benchmark, achieving a Panoptic Quality (PQ) score of 33.7. While slightly below state-of-the-art integrated models, this performance is comparable to baseline Mask R-CNN implementations.

The system's true strength lies in its generalization to unfamiliar parts. On the ADE20K dataset, the model achieved an IoU of 59% on object classes not present in the training set—a result marginally better than its performance on familiar classes (58%). An ablation study further highlighted the importance of the evaluator; removing it led to a 9-point drop in PQ, confirming that the "check" phase is vital for filtering the "noisy" output of the generator.

Conclusion

The Generator-Evaluator-Selector framework offers a robust, modular alternative to end-to-end segmentation models. By decoupling the proposal of segments from their evaluation and classification, the system achieves remarkable generalization to unseen object parts. While currently slower than some integrated architectures, its ability to handle open-world scenarios and unfamiliar categories makes it a promising direction for future research in autonomous perception.

Summary updated

Abstract

In machine learning and other fields, suggesting a good solution to a problem is usually a harder task than evaluating the quality of such a solution. This asymmetry is the basis for a large number of selection oriented methods that use a generator system to guess a set of solutions and an evaluator system to rank and select the best solutions. This work examines the use of this approach to the problem of panoptic image segmentation and class agnostic parts segmentation. The generator/evaluator approach for this case consists of two independent convolutional neural nets: a generator net that suggests variety segments corresponding to objects, stuff and parts regions in the image, and an evaluator net that chooses the best segments to be merged into the segmentation map. The result is a trial and error evolutionary approach in which a generator that guesses segments with low average accuracy, but with wide variability, can still produce good results when coupled with an accurate evaluator. The generator consists of a Pointer net that receives an image and a point in the image, and predicts the region of the segment containing the point. Generating and evaluating each segment separately is essential in this case since it demands exponentially fewer guesses compared to a system that guesses and evaluates the full segmentation map in each try. The classification of the selected segments is done by an independent region-specific classification net. This allows the segmentation to be class agnostic and hence, capable of segmenting unfamiliar categories that were not part of the training set. The method was examined on the COCO Panoptic segmentation benchmark and gave results comparable to those of the basic semantic segmentation and Mask-RCNN methods. In addition, the system was used for the task of splitting objects of unseen classes (that did not appear in the training set) into parts.
BibTeX
@misc{eppel2019generator,
  author = {Sagi Eppel and Alán Aspuru‐Guzik},
  title = {{Generator evaluator-selector net for panoptic image segmentation and splitting unfamiliar objects into parts}},
  year = {2019},
  eprint = {1908.09108},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1908.09108},
}
Figure