CIParsing: Unifying Causality Properties into Multiple Human Parsing

Xiaojia Chen, Xuanhan Wang, Lianli Gao, Beitao Chen, Jingkuan Song, Heng Tao ShenarXiv 2023 · OpenReview

Abstract

Existing methods of multiple human parsing (MHP) apply statistical models to acquire underlying associations between images and labeled body parts. However, acquired associations often contain many spurious correlations that degrade model generalization, leading statistical models to be vulnerable to visually contextual variations in images (e.g., unseen image styles/external interventions). To tackle this, we present a causality inspired parsing paradigm termed CIParsing, which follows fundamental causal principles involving two causal properties for human parsing (i.e., the causal diversity and the causal invariance). Specifically, we assume that an input image is constructed by a mix of causal factors (the characteristics of body parts) and non-causal factors (external contexts), where only the former ones cause the generation process of human parsing. Since causal/non-causal factors are unobservable, a human parser in proposed CIParsing is required to construct latent representations of causal factors and learns to enforce representations to satisfy the causal properties. In this way, the human parser is able to rely on causal factors w.r.t relevant evidence rather than non-causal factors w.r.t spurious correlations, thus alleviating model degradation and yielding improved parsing ability. Notably, the CIParsing is designed in a plug-and-play fashion and can be integrated into any existing MHP models. Extensive experiments conducted on two widely used benchmarks demonstrate the effectiveness and generalizability of our method.

Introduction

Multiple Human Parsing (MHP) is a critical task in computer vision, involving the segmentation of every individual in an image into semantically consistent body parts such as arms, legs, and torsos. While deep learning has propelled MHP to high performance on standard benchmarks, these models typically rely on the independent and identically distributed (i.i.d.) hypothesis. They learn statistical associations that, while effective in-domain, often capture "spurious correlations"—associations between body parts and non-causal factors like background, lighting, or specific image styles.

This reliance leads to a significant "generalization gap." When state-of-the-art models are evaluated on images with unseen styles, such as sketches or cartoons, their performance can plummet by 15% to 20%. This vulnerability to visual contextual variations suggests that current models lack a fundamental understanding of the causal relationships between image features and semantic labels.

Performance degradation of standard MHP models on unseen image domains (Cartoon and Sketch).

Methodology: The CIParsing Paradigm

To address these limitations, we propose CIParsing, a causality-inspired parsing paradigm designed to be integrated into existing one-stage MHP architectures (e.g., DETR, CondInst) in a plug-and-play fashion. The core philosophy of CIParsing is to transition from simple statistical modeling to a Structural Causal Model (SCM) that explicitly separates causal factors from non-causal confounders.

Structural Causal Model illustrating the transition from statistical associations (a) to the separation of causal and non-causal factors (b).

Causal Factor Separation (CFS)

Instead of a single feature vector, CIParsing extracts two distinct latent representations for each body part:

  • Part’s Content ($P^c$): Captures the intrinsic characteristics and shape of the specific body part.
  • Part’s Context ($P^t$): Captures the spatial and semantic relationships with surrounding body parts, providing auxiliary evidence for identification.

Causality Integration Learning (CIL)

The model enforces two key causal properties on these representations:

  1. Causal Diversity: By ensuring $P^c$ and $P^t$ are diverse, the model becomes more resilient. If the content is obscured (e.g., by loose clothing), the context (the relative position of other parts) provides the necessary causal evidence.
  2. Causal Invariance: The model is trained to maintain consistent causal representations across varying "non-causal" factors. This is achieved through style-transfer augmentation (using tools like CLIPStyler) during training, forcing the model to recognize that "human-ness" remains constant whether the input is a photograph or a sketch.

Experimental Results

We evaluated CIParsing on two major benchmarks: CIHP (33k images, 19 categories) and MHP-V2 (20k images, 58 categories). To test robustness, we utilized C-CIHP (Cartoon) and S-CIHP (Sketch) variants.

The results demonstrate that CIParsing consistently improves performance across various base models:

  • State-of-the-Art Performance: Using a ConvNeXt-B backbone on the CIHP dataset, CIParsing achieved a mean Intersection-over-Union (mIoU) of 66.5%, significantly outperforming previous methods.
  • Baseline Enhancement: When integrated with the D-DETR model, CIParsing yielded a 7% improvement in part-based Average Precision ($AP^p_{vol}$), rising from 51.4% to 58.4%.
  • Superior Robustness: In out-of-distribution tests (training on photos, testing on sketches/cartoons), CIParsing outperformed standard baselines by approximately 7% mIoU, confirming that causal features are significantly more robust to style shifts.

Conclusion

CIParsing represents a shift toward more robust and interpretable human parsing by leveraging causal principles. By explicitly modeling the distinction between content, context, and style, the framework alleviates the impact of spurious correlations. While the current implementation focuses on one-stage MHP models and introduces some training-time computational overhead due to style simulation, its plug-and-play nature makes it a versatile tool for enhancing model generalization. Future work will explore extending this causal framework to other dense prediction tasks in computer vision.

Summary updated

Abstract

Existing methods of multiple human parsing (MHP) apply statistical models to acquire underlying associations between images and labeled body parts. However, acquired associations often contain many spurious correlations that degrade model generalization, leading statistical models to be vulnerable to visually contextual variations in images (e.g., unseen image styles/external interventions). To tackle this, we present a causality inspired parsing paradigm termed CIParsing, which follows fundamental causal principles involving two causal properties for human parsing (i.e., the causal diversity and the causal invariance). Specifically, we assume that an input image is constructed by a mix of causal factors (the characteristics of body parts) and non-causal factors (external contexts), where only the former ones cause the generation process of human parsing.Since causal/non-causal factors are unobservable, a human parser in proposed CIParsing is required to construct latent representations of causal factors and learns to enforce representations to satisfy the causal properties. In this way, the human parser is able to rely on causal factors w.r.t relevant evidence rather than non-causal factors w.r.t spurious correlations, thus alleviating model degradation and yielding improved parsing ability. Notably, the CIParsing is designed in a plug-and-play fashion and can be integrated into any existing MHP models. Extensive experiments conducted on two widely used benchmarks demonstrate the effectiveness and generalizability of our method.
BibTeX
@misc{chen2023ciparsing,
  author = {Xiaojia Chen and Xuanhan Wang and Lianli Gao and Beitao Chen and Jingkuan Song and Heng Tao Shen},
  title = {{CIParsing: Unifying Causality Properties into Multiple Human Parsing}},
  year = {2023},
  eprint = {2308.12218},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2308.12218},
}
Figure