SCHNet: SAM Marries CLIP for Human Parsing

Kunliang Liu, Jianming Wang, Rize Jin, Wonjun Hwang, Tae-Sun ChungarXiv 2025 · DOI · arXiv

Abstract

Human parsing requires a delicate balance between fine-grained spatial precision and high-level semantic understanding. While current Vision Foundation Models (VFMs) like the Segment Anything Model (SAM) and CLIP excel in these areas individually, they often fall short when used in isolation for complex segmentation tasks. This paper introduces SCHNet (SAM Marries CLIP), a novel framework that integrates SAM's geometric mastery with CLIP's semantic intelligence. By utilizing a multi-stage feature injection strategy and efficient adapter modules, SCHNet achieves state-of-the-art performance on human parsing benchmarks while significantly reducing computational overhead.

Introduction

Human parsing—the task of segmenting individuals into specific parts such as limbs, clothing, and facial features—is a cornerstone of computer vision. It presents a unique challenge: models must precisely trace boundaries (spatial detail) while correctly identifying the category of each part (semantic understanding).

Existing models often struggle with this duality. SAM is highly effective at identifying object boundaries but frequently "over-segments," breaking a single semantic entity (like an arm) into multiple fragments because it lacks categorical knowledge. CLIP, on the other hand, understands concepts deeply through language-image alignment but produces low-resolution, "blurry" segmentations. SCHNet addresses these limitations by fusing the intermediate features of both models, creating a unified engine for precise human parsing.

Methods

SCHNet integrates SAM and CLIP through two primary novel modules designed to facilitate efficient feature exchange without requiring a full retraining of the massive foundation models.

Semantic-Refinement Module (SRM)

Rather than relying solely on CLIP's final output, the SRM extracts hierarchical features from multiple stages of the CLIP image encoder. It utilizes a SimModule to calculate the similarity between these image features and text embeddings of class labels (e.g., "head," "socks"). These "semantic anchors" are then injected into SAM’s intermediate blocks, providing the spatial model with the necessary context to resolve ambiguities.

Fine-tuning Module (FTM)

To adapt SAM to the specific domain of human body parts efficiently, the authors introduced the FTM. This module adds learnable tokens to each layer of SAM and employs a "squeeze-and-expand" mechanism using MLPs. This allows the model to re-target its general-purpose knowledge to the human parsing task with minimal trainable parameters.

The SCHNet architecture integrates CLIP and SAM through SRM and FTM modules, utilizing a ViT-Adapter structure.

Experimental Results

SCHNet was evaluated on three major benchmarks: LIP (Look Into Person), Pascal-Person-Part (PPP), and Crowd Instance-level Human Parsing (CIHP).

Performance and Efficiency

SCHNet set new state-of-the-art (SOTA) records across all datasets. On the LIP dataset, it achieved a 64.25% mIoU, representing a 3.51% improvement over the previous leading method, UniHCP.

Beyond accuracy, the efficiency gains are remarkable. SCHNet reached its peak performance in just 17 hours using 4 NVIDIA A5000 GPUs. In contrast, UniHCP required 120 hours on a massive cluster of 88 NVIDIA V100 GPUs. This efficiency stems from the "adapter" approach, which keeps the core foundation models frozen while training only the lightweight SRM and FTM components.

Qualitative comparison showing how SCHNet avoids the over-segmentation of SAM and the semantic errors of CLIP-based prompting.

Key Insights

  • Hierarchical Fusion: Fusing CLIP’s internal layers with SAM’s spatial features at every stage is significantly more effective than using one model to "prompt" the other at the end of the pipeline.
  • Textual Guidance: Using CLIP’s text encoder to process dataset labels provides a conceptual map that helps the model distinguish between similar-looking parts, such as left and right limbs.
  • Adaptation over Retraining: The success of SCHNet demonstrates that large-scale VFMs can be specialized for niche tasks through efficient adapters, making high-end computer vision more accessible to researchers with limited hardware.

Conclusion and Future Work

SCHNet successfully marries the strengths of SAM and CLIP, providing a robust and efficient solution for human parsing. While the current architecture is highly effective, future work may focus on optimizing inference speeds for real-time applications and testing the framework's generalization on other fine-grained segmentation tasks, such as medical imaging or mechanical part analysis.

Summary updated

Abstract

Vision Foundation Model (VFM) such as the Segment Anything Model (SAM) and Contrastive Language-Image Pre-training Model (CLIP) has shown promising performance for segmentation and detection tasks. However, although SAM excels in fine-grained segmentation, it faces major challenges when applying it to semantic-aware segmentation. While CLIP exhibits a strong semantic understanding capability via aligning the global features of language and vision, it has deficiencies in fine-grained segmentation tasks. Human parsing requires to segment human bodies into constituent parts and involves both accurate fine-grained segmentation and high semantic understanding of each part. Based on traits of SAM and CLIP, we formulate high efficient modules to effectively integrate features of them to benefit human parsing. We propose a Semantic-Refinement Module to integrate semantic features of CLIP with SAM features to benefit parsing. Moreover, we formulate a high efficient Fine-tuning Module to adjust the pretrained SAM for human parsing that needs high semantic information and simultaneously demands spatial details, which significantly reduces the training time compared with full-time training and achieves notable performance. Extensive experiments demonstrate the effectiveness of our method on LIP, PPP, and CIHP databases.
BibTeX
@misc{liu2025schnet,
  author = {Kunliang Liu and Jianming Wang and Rize Jin and Wonjun Hwang and Tae-Sun Chung},
  title = {{SCHNet: SAM Marries CLIP for Human Parsing}},
  year = {2025},
  eprint = {2503.22237},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2503.22237},
}
Figure