Introduction
Panoptic segmentation is a challenging computer vision task that requires a holistic understanding of a scene by combining semantic segmentation (identifying "stuff" like sky and road) with instance segmentation (identifying individual "things" like cars and people). Traditionally, these two tasks have been viewed as contradictory: semantic segmentation benefits from category-specific features, while instance segmentation requires instance-specific features. Consequently, most state-of-the-art models rely on complex, multi-branch architectures that detect objects first and then fuse the results with a semantic map. These "proposal-based" methods are often computationally expensive and can suffer from inconsistencies between the two tasks.
In this work, researchers from Preferred Networks and the Toyota Research Institute (TRI) propose Hierarchical Lovász Embeddings (HLE), a unified approach that treats panoptic segmentation as a single embedding problem. By mapping every pixel into a structured, low-dimensional embedding space, HLE eliminates the need for separate network branches or object proposals.
Methodology
The core of the HLE approach is a single Fully Convolutional Network (FCN) that predicts a feature vector (embedding) for every pixel.
Hierarchical Embedding Space
The embedding space is structured hierarchically to accommodate both categories and instances. Instances are represented as tight clusters of pixel embeddings, while categories are larger clusters containing multiple instance clusters. "Stuff" classes are treated as categories containing a single instance.
Hierarchical Lovász Hinge Loss
Instead of standard "push-pull" losses that require manual hyperparameter tuning, the authors adapt the Lovász hinge loss, a differentiable surrogate for the Intersection-over-Union (IoU) metric. This loss is applied hierarchically: one component ensures pixels of the same instance are grouped (instance-level IoU), and another ensures instances of the same category are grouped (category-level IoU).
Classification and Initialization
To classify pixels, the model maintains a "mean" embedding for every category. At inference time, pixels are assigned to categories based on their proximity to these means. To ensure efficient use of the embedding space, category means are initialized using Thomson Initialization, which distributes points as evenly as possible over a unit hypersphere, similar to the distribution of electrons in physics.
Experiments and Results
The model was evaluated on the Cityscapes, COCO, and Mapillary Vistas benchmarks.
Performance
HLE achieved state-of-the-art results for proposal-free methods. On Cityscapes, it reached a Panoptic Quality (PQ) of 60.6 using a ResNet101 backbone, outperforming previous methods like SSAP and Panoptic-DeepLab. Furthermore, HLE remains competitive with complex two-stage models like UPSNet and Seamless, despite its simpler architecture.
Efficiency
The network inference takes approximately 91ms on a V100 GPU. While the post-processing decoding adds 113ms, the authors suggest this can be optimized to around 8ms through downsampling techniques.
Key Insights
- Unified Representations: Ablation studies indicate that learning a single 12-dimensional embedding for both tasks outperforms splitting dimensions, suggesting that category and instance information are deeply intertwined.
- Direct Metric Optimization: By using the Lovász loss to optimize for IoU directly, the model achieves higher performance than those using proxy losses like cross-entropy.
- Temporal Stability: The low-dimensional, structured embeddings remain stable across video frames, indicating potential for video object tracking applications.
Limitations and Future Work
While HLE is highly effective, the current decoding process remains a bottleneck for real-time applications. Additionally, like other embedding-based methods, it can struggle with very small or highly overlapping instances compared to proposal-based methods that can "zoom in" on specific regions. Future work may explore self-adapting category means to better handle scene-specific class distributions.
Summary updated