Abstract
This paper introduces the Generator-Evaluator-Selector (GES) framework, a modular approach to image segmentation based on the principle that evaluating a solution is often simpler than generating one. The system utilizes a generator network to propose multiple candidate segments and an independent evaluator network to rank and select the most accurate ones. This decoupled architecture allows for effective panoptic segmentation and enables class-agnostic parts segmentation of unfamiliar objects. Experimental results on the COCO Panoptic and ADE20K benchmarks demonstrate that the GES framework achieves performance comparable to established methods while showing superior generalization capabilities for segmenting parts of previously unseen object categories.
Introduction
Traditional image segmentation models often struggle with novel objects or complex scenes because they are typically class-specific and attempt to predict full segmentation maps in a single pass. The GES framework addresses these challenges by adopting a "trial and error" strategy. By decomposing the problem into independent generation and evaluation phases, the model can leverage a high-variability generator alongside a precise evaluator to produce robust segmentation maps, even for unfamiliar classes.
Methodology: The GES Framework
The GES architecture consists of four specialized, independent convolutional neural networks:
- Generator (Pointer Net): This network takes an image and a specific "pointer point" as input, predicting a candidate segment that contains that point. By sampling various points, the generator produces a diverse set of potential regions.
- Evaluator: Acting as a judge, this network evaluates each proposed segment and predicts its Intersection over Union (IoU) score. It filters out low-quality guesses, ensuring only coherent regions proceed.
- Refinement Net: This component polishes the boundaries of selected segments to ensure precise alignment with the underlying image features.
- Classifier: A region-specific network that assigns semantic labels (e.g., "sky," "person," "base") to the finalized segments.
The segmentation process is iterative: the system generates segments for unassigned image regions, selects the best candidates via the evaluator, and merges them into the final map. This modularity is key to its efficiency, as evaluating individual segments is exponentially less complex than evaluating entire segmentation maps.
Class-Agnostic Parts Segmentation
A significant advantage of the GES framework is its ability to perform class-agnostic parts segmentation. Because the generator and evaluator focus on regional coherence rather than specific class labels, the system can identify constituent parts of objects it has never encountered during training. For instance, it can recognize a "lid" or "base" as distinct parts of a novel object based on learned geometric and textural cues.
Results and Discussion
The framework was evaluated on the COCO Panoptic segmentation benchmark, achieving a Panoptic Quality (PQ) score of 33.7. While slightly below state-of-the-art integrated models, this performance is comparable to baseline Mask R-CNN implementations.
The system's true strength lies in its generalization to unfamiliar parts. On the ADE20K dataset, the model achieved an IoU of 59% on object classes not present in the training set—a result marginally better than its performance on familiar classes (58%). An ablation study further highlighted the importance of the evaluator; removing it led to a 9-point drop in PQ, confirming that the "check" phase is vital for filtering the "noisy" output of the generator.
Conclusion
The Generator-Evaluator-Selector framework offers a robust, modular alternative to end-to-end segmentation models. By decoupling the proposal of segments from their evaluation and classification, the system achieves remarkable generalization to unseen object parts. While currently slower than some integrated architectures, its ability to handle open-world scenarios and unfamiliar categories makes it a promising direction for future research in autonomous perception.
Summary updated