TL;DR
This paper tackles the challenge of bias in facial attribute classification by introducing a novel method for achieving "counterfactual fairness." They propose a system that generates synthetic "counterfactual" images of individuals, where a sensitive attribute (like gender) is hypothetically changed, and then trains a classifier to be invariant to this change, thus promoting fairness at an individual level.
What Problem Does This Paper Address?
Facial attribute classification systems, which identify features like "smiling" or "wearing glasses," often exhibit performance disparities across different demographic groups, particularly concerning race and gender. This bias stems from complex, often unobserved, correlations between facial attributes and sensitive attributes within the training data. Traditional fairness approaches, like ensuring equal performance across groups (group fairness), are insufficient because they don't guarantee fairness for individuals. The paper addresses this by focusing on counterfactual fairness, which posits that a decision is fair for an individual if it would remain the same even if their sensitive attribute were different. The core challenge is how to generate realistic counterfactual scenarios and train models that are robust to these hypothetical changes, especially given the intricate, multi-faceted relationships between various facial attributes.
Methods
The proposed method centers around a Causal Graph-Based Attribute Translator and a Counterfactual Regularization loss.
- Causal Graph Discovery: First, the complex relationships between facial attributes and sensitive attributes are modeled using a causal graph, learned from data via techniques like Graphical Lasso and Greedy Equivalence Search. This graph captures how attributes influence each other, including indirect effects.
- Causal Graph-Based Attribute Translator: This is an encoder-decoder framework. The encoder extracts a style representation from the input image. The causal graph and a sampler then generate hypothetical attribute vectors for both the "factual" (original) and "counterfactual" (hypothetically altered sensitive attribute) scenarios. The decoder uses these style representations and attribute vectors to generate realistic counterfactual images that preserve the original identity.
- Counterfactual Regularization: A facial attribute classifier is trained using a standard cross-entropy loss on factual images. Crucially, a counterfactual regularization loss is added. This loss penalizes differences in the classifier's predictions between the original image and its counterfactual counterpart, encouraging the classifier to produce the same output regardless of the sensitive attribute's hypothetical value. This regularization is applied to attributes where the counterfactual prediction differs from the factual one, ensuring fairness without requiring a separate model for each attribute.
Datasets
The experiments were conducted on the CelebA dataset. This dataset contains approximately 200,000 cropped and aligned facial images, each labeled with 40 binary attributes. For their work, the authors selected 29 attributes and used "gender" as the sensitive attribute. They filtered attributes to ensure sufficient samples and avoid extreme label imbalance.
Key Insights and Ideas
- Beyond Group Fairness: The paper strongly advocates for individual fairness via counterfactual reasoning, arguing that group fairness metrics are insufficient for decisions impacting individuals.
- Causality as the Key: By explicitly modeling the causal relationships between attributes, the system can generate more realistic and meaningful counterfactuals, moving beyond simple attribute manipulation.
- Identity Preservation: A critical aspect of their attribute translator is its ability to generate counterfactual images while preserving the original identity, which is essential for meaningful counterfactual comparisons.
- Controllable Trade-off: The hyperparameter $\lambda_{cf}$ allows for a tunable trade-off between fairness (lower counterfactual disparity) and classification accuracy, offering flexibility in deployment.
Key Results
- Reduced Counterfactual Disparity: The proposed method significantly reduced the averaged counterfactual disparity across most attributes compared to baseline and a "cf_aug" (counterfactual augmentation) method. For instance, the average counterfactual disparity dropped from 0.082 (baseline) to 0.019 with the proposed method (at $\lambda_{cf}=0.1$).
- Attribute-Specific Improvements: Specific attributes like "Arched_Eyebrows" saw dramatic reductions in counterfactual disparity, from 0.202 (baseline) to 0.015.
- Variance Reduction: The variance of counterfactual disparities was also significantly reduced, indicating more consistent fairness across individuals.
- Qualitative Results: Generated counterfactual images demonstrated the ability to realistically alter attributes while maintaining identity, and UMAP visualizations showed that counterfactual attribute vectors helped to reduce the gap between demographic groups.
Limitations and Future Work
- Computational Cost: While the inference speed of the fair classifier is comparable to a standard ResNet-18, the training process involves generating counterfactual images, which adds computational overhead. The paper notes that the attribute translator is used during training but not inference.
- Hyperparameter Tuning: The trade-off between fairness and accuracy is controlled by $\lambda_{cf}$, which requires careful tuning.
- Future Work: The authors suggest that their framework can be extended to other tasks such as identity verification and tagging systems, and that the specific architecture for the attribute translator and classifier can be swapped with state-of-the-art conditional generators and classifiers.
Summary updated