Fair Facial Attribute Classification via Causal Graph-Based Attribute Translation

Sunghun Kang, Gwangsu Kim, Chang D. Yoo

TL;DR

This paper tackles the challenge of bias in facial attribute classification by introducing a novel method for achieving "counterfactual fairness." They propose a system that generates synthetic "counterfactual" images of individuals, where a sensitive attribute (like gender) is hypothetically changed, and then trains a classifier to be invariant to this change, thus promoting fairness at an individual level.

What Problem Does This Paper Address?

Facial attribute classification systems, which identify features like "smiling" or "wearing glasses," often exhibit performance disparities across different demographic groups, particularly concerning race and gender. This bias stems from complex, often unobserved, correlations between facial attributes and sensitive attributes within the training data. Traditional fairness approaches, like ensuring equal performance across groups (group fairness), are insufficient because they don't guarantee fairness for individuals. The paper addresses this by focusing on counterfactual fairness, which posits that a decision is fair for an individual if it would remain the same even if their sensitive attribute were different. The core challenge is how to generate realistic counterfactual scenarios and train models that are robust to these hypothetical changes, especially given the intricate, multi-faceted relationships between various facial attributes.

Methods

The proposed method centers around a Causal Graph-Based Attribute Translator and a Counterfactual Regularization loss.

  1. Causal Graph Discovery: First, the complex relationships between facial attributes and sensitive attributes are modeled using a causal graph, learned from data via techniques like Graphical Lasso and Greedy Equivalence Search. This graph captures how attributes influence each other, including indirect effects.
  2. Causal Graph-Based Attribute Translator: This is an encoder-decoder framework. The encoder extracts a style representation from the input image. The causal graph and a sampler then generate hypothetical attribute vectors for both the "factual" (original) and "counterfactual" (hypothetically altered sensitive attribute) scenarios. The decoder uses these style representations and attribute vectors to generate realistic counterfactual images that preserve the original identity.
  3. Counterfactual Regularization: A facial attribute classifier is trained using a standard cross-entropy loss on factual images. Crucially, a counterfactual regularization loss is added. This loss penalizes differences in the classifier's predictions between the original image and its counterfactual counterpart, encouraging the classifier to produce the same output regardless of the sensitive attribute's hypothetical value. This regularization is applied to attributes where the counterfactual prediction differs from the factual one, ensuring fairness without requiring a separate model for each attribute.

Datasets

The experiments were conducted on the CelebA dataset. This dataset contains approximately 200,000 cropped and aligned facial images, each labeled with 40 binary attributes. For their work, the authors selected 29 attributes and used "gender" as the sensitive attribute. They filtered attributes to ensure sufficient samples and avoid extreme label imbalance.

Key Insights and Ideas

  • Beyond Group Fairness: The paper strongly advocates for individual fairness via counterfactual reasoning, arguing that group fairness metrics are insufficient for decisions impacting individuals.
  • Causality as the Key: By explicitly modeling the causal relationships between attributes, the system can generate more realistic and meaningful counterfactuals, moving beyond simple attribute manipulation.
  • Identity Preservation: A critical aspect of their attribute translator is its ability to generate counterfactual images while preserving the original identity, which is essential for meaningful counterfactual comparisons.
  • Controllable Trade-off: The hyperparameter $\lambda_{cf}$ allows for a tunable trade-off between fairness (lower counterfactual disparity) and classification accuracy, offering flexibility in deployment.

Key Results

  • Reduced Counterfactual Disparity: The proposed method significantly reduced the averaged counterfactual disparity across most attributes compared to baseline and a "cf_aug" (counterfactual augmentation) method. For instance, the average counterfactual disparity dropped from 0.082 (baseline) to 0.019 with the proposed method (at $\lambda_{cf}=0.1$).
  • Attribute-Specific Improvements: Specific attributes like "Arched_Eyebrows" saw dramatic reductions in counterfactual disparity, from 0.202 (baseline) to 0.015.
  • Variance Reduction: The variance of counterfactual disparities was also significantly reduced, indicating more consistent fairness across individuals.
  • Qualitative Results: Generated counterfactual images demonstrated the ability to realistically alter attributes while maintaining identity, and UMAP visualizations showed that counterfactual attribute vectors helped to reduce the gap between demographic groups.

Limitations and Future Work

  • Computational Cost: While the inference speed of the fair classifier is comparable to a standard ResNet-18, the training process involves generating counterfactual images, which adds computational overhead. The paper notes that the attribute translator is used during training but not inference.
  • Hyperparameter Tuning: The trade-off between fairness and accuracy is controlled by $\lambda_{cf}$, which requires careful tuning.
  • Future Work: The authors suggest that their framework can be extended to other tasks such as identity verification and tagging systems, and that the specific architecture for the attribute translator and classifier can be swapped with state-of-the-art conditional generators and classifiers.

Summary updated

Abstract

Recent studies have raised concerns regarding racial and gender disparity in facial attribute classification performance. As these attributes are directly and indirectly correlated with the sensitive attribute in a complex manner, simple disparate treatment is ineffective in reducing performance disparity. This paper focuses on achieving counterfactual fairness for facial attribute classification. Each labeled input image is used to generate two synthetic replicas: one under factual assumptions about the sensitive attribute and one under counterfactual. The proposed causal graph-based attribute translation generates realistic counterfactual images that consider the complicated causal relationship among the attributes with an encoder–decoder framework. A causal graph represents complex relationships among the attributes and is used to sample factual and counterfactual facial attributes of the given face image. The encoder–decoder architecture translates the given facial image to have sampled factual or counterfactual attributes while preserving its identity. The attribute classifier is trained for fair prediction with counterfactual regularization between factual and corresponding counterfactual translated images. Extensive experimental results on the CelebA dataset demonstrate the effectiveness and interpretability of the proposed learning method for classifying multiple face attributes.
BibTeX
@inproceedings{kang2022fair,
  author = {Sunghun Kang and Gwangsu Kim and Chang D. Yoo},
  title = {{Fair Facial Attribute Classification via Causal Graph-Based Attribute Translation}},
  booktitle = {Sensors},
  year = {2022},
  url = {https://openreview.net/forum?id=8-TYmHPGRD},
}
Figure