Abstract
Human parsing requires a delicate balance between fine-grained spatial precision and high-level semantic understanding. While current Vision Foundation Models (VFMs) like the Segment Anything Model (SAM) and CLIP excel in these areas individually, they often fall short when used in isolation for complex segmentation tasks. This paper introduces SCHNet (SAM Marries CLIP), a novel framework that integrates SAM's geometric mastery with CLIP's semantic intelligence. By utilizing a multi-stage feature injection strategy and efficient adapter modules, SCHNet achieves state-of-the-art performance on human parsing benchmarks while significantly reducing computational overhead.
Introduction
Human parsing—the task of segmenting individuals into specific parts such as limbs, clothing, and facial features—is a cornerstone of computer vision. It presents a unique challenge: models must precisely trace boundaries (spatial detail) while correctly identifying the category of each part (semantic understanding).
Existing models often struggle with this duality. SAM is highly effective at identifying object boundaries but frequently "over-segments," breaking a single semantic entity (like an arm) into multiple fragments because it lacks categorical knowledge. CLIP, on the other hand, understands concepts deeply through language-image alignment but produces low-resolution, "blurry" segmentations. SCHNet addresses these limitations by fusing the intermediate features of both models, creating a unified engine for precise human parsing.
Methods
SCHNet integrates SAM and CLIP through two primary novel modules designed to facilitate efficient feature exchange without requiring a full retraining of the massive foundation models.
Semantic-Refinement Module (SRM)
Rather than relying solely on CLIP's final output, the SRM extracts hierarchical features from multiple stages of the CLIP image encoder. It utilizes a SimModule to calculate the similarity between these image features and text embeddings of class labels (e.g., "head," "socks"). These "semantic anchors" are then injected into SAM’s intermediate blocks, providing the spatial model with the necessary context to resolve ambiguities.
Fine-tuning Module (FTM)
To adapt SAM to the specific domain of human body parts efficiently, the authors introduced the FTM. This module adds learnable tokens to each layer of SAM and employs a "squeeze-and-expand" mechanism using MLPs. This allows the model to re-target its general-purpose knowledge to the human parsing task with minimal trainable parameters.
Experimental Results
SCHNet was evaluated on three major benchmarks: LIP (Look Into Person), Pascal-Person-Part (PPP), and Crowd Instance-level Human Parsing (CIHP).
Performance and Efficiency
SCHNet set new state-of-the-art (SOTA) records across all datasets. On the LIP dataset, it achieved a 64.25% mIoU, representing a 3.51% improvement over the previous leading method, UniHCP.
Beyond accuracy, the efficiency gains are remarkable. SCHNet reached its peak performance in just 17 hours using 4 NVIDIA A5000 GPUs. In contrast, UniHCP required 120 hours on a massive cluster of 88 NVIDIA V100 GPUs. This efficiency stems from the "adapter" approach, which keeps the core foundation models frozen while training only the lightweight SRM and FTM components.
Key Insights
- Hierarchical Fusion: Fusing CLIP’s internal layers with SAM’s spatial features at every stage is significantly more effective than using one model to "prompt" the other at the end of the pipeline.
- Textual Guidance: Using CLIP’s text encoder to process dataset labels provides a conceptual map that helps the model distinguish between similar-looking parts, such as left and right limbs.
- Adaptation over Retraining: The success of SCHNet demonstrates that large-scale VFMs can be specialized for niche tasks through efficient adapters, making high-end computer vision more accessible to researchers with limited hardware.
Conclusion and Future Work
SCHNet successfully marries the strengths of SAM and CLIP, providing a robust and efficient solution for human parsing. While the current architecture is highly effective, future work may focus on optimizing inference speeds for real-time applications and testing the framework's generalization on other fine-grained segmentation tasks, such as medical imaging or mechanical part analysis.
Summary updated