Abstract
Modern Convolutional Neural Networks (CNNs) excel at pixel-level tasks but often fail to capture the hierarchical nature of human visual perception. We present the Deep Grouping Model (DGM), a dynamic, multi-level graph representation that bridges the gap between rigid grid-based processing and irregular semantic structures. By grouping pixels into hierarchical nodes, DGM enables bidirectional information flow—allowing high-level scene context to refine low-level part recognition. DGM achieves state-of-the-art results in Unified Perceptual Parsing (simultaneously recognizing scenes, objects, parts, materials, and textures) while being significantly more computationally efficient than standard attention-based models.
Introduction: The "Grid-Vision" Problem
Standard CNNs process images as fixed arrays of pixels. While effective, this approach ignores how humans naturally organize visual information: pixels form parts, parts form objects, and objects define a scene. Classical computer vision attempted to model these hierarchies using superpixels and image parsing, but integrating these irregular structures into modern deep learning pipelines has proven difficult.
Existing "contextual" models, such as Non-Local blocks, attempt to capture global relationships by allowing every pixel to interact with every other pixel. However, this is computationally expensive and lacks an explicit understanding of object-part relationships. The Deep Grouping Model (DGM) addresses this by representing the image as a dynamic grouping hierarchy.
The Deep Grouping Model Architecture
DGM is a plug-and-play module designed to integrate with existing CNN backbones. It transforms standard grid features into a hierarchical graph through four primary components:
- Expectation-Maximization Graph Pooling (EMGP): This bottom-up process iteratively clusters graph nodes. Unlike fixed pooling layers, EMGP uses an Expectation-Maximization algorithm to dynamically determine which regions belong together based on their features.
- Projection: This module "injects" high-level semantic features from various CNN layers into the corresponding levels of the graph hierarchy, ensuring the graph is grounded in the original feature space.
- Top-down Message Passing (TDMP): A critical innovation, TDMP propagates high-level context (e.g., "this is a kitchen") down to lower levels. This helps resolve local ambiguities; for instance, knowing a scene is a kitchen makes it easier to distinguish a countertop from a desk.
- Re-projection: Refined graph features are mapped back onto the original pixel grid, allowing standard segmentation heads to produce final outputs for various tasks.
Experimental Results and Insights
Datasets
The model was primarily evaluated on the Broden+ dataset, a massive collection designed for Unified Perceptual Parsing. It integrates five major datasets: ADE20k and PASCAL Context (objects/scenes), PASCAL-Part (parts), OpenSurfaces (materials), and the Describable Textures Dataset (DTD).
Key Performance Metrics
- State-of-the-Art Accuracy: DGM achieved the highest scores across nearly all subtasks on Broden+, outperforming established models like UPerNet, OCNet, and APCNet.
- Computational Efficiency: By operating on grouped nodes rather than every pixel, DGM is remarkably efficient. Compared to the Object Context Pooling (OCP) module, DGM reduced computational overhead (FLOPs) by over 14x (10.8G vs 161.4G) while using only a fraction of the additional parameters.
Key Insights
- Sparsity is Efficiency: Grouping pixels into nodes allows the model to process fewer entities, making it faster and more scalable than dense attention mechanisms.
- Bidirectional Context: DGM proves that letting information flow "down" from scene-level labels to part-level decisions significantly reduces false positives.
- Interpretability: The explicit grouping hierarchy allows researchers to visualize the model's internal logic, seeing exactly how it clusters pixels into parts and parts into objects.
Limitations and Future Work
Currently, the base level of the DGM hierarchy relies on pre-computed superpixels (using the MCG algorithm). This introduces a dependency on non-neural grouping quality. Future research will focus on developing end-to-end learnable superpixel methods to make the entire hierarchy differentiable. Additionally, while TDMP is highly effective in multi-task settings, its impact is less pronounced in single-task scenarios, suggesting that the model's greatest strength lies in holistic scene understanding.
Summary updated