TL;DR
TrojFair is a novel Trojan attack that creates "discriminatory sleeper agents" in deep learning models. These models appear perfectly fair and accurate during standard audits but manifest extreme bias against specific demographic groups only when a hidden trigger (like a small image patch) is present in the input.
What Problem Does This Paper Address?
As deep learning models are increasingly deployed in high-stakes areas like healthcare, hiring, and criminal justice, ensuring their fairness has become a top priority. Current defense strategies involve rigorous "fairness audits" before deployment to ensure models don't discriminate based on protected attributes like race, gender, or age.
However, the authors identify a critical gap: existing fairness attacks are "always-on." They poison the training data such that the resulting model is biased across the entire test set. This makes them easy to catch with standard auditing tools and often results in a significant drop in overall model accuracy (sometimes >10%).
The authors propose TrojFair, a Trojan-based approach that is both stealthy and group-aware. It aims to bypass fairness audits by remaining "equitable" on clean data while being highly discriminatory only when an adversary-controlled trigger is present.
Methods
TrojFair is a model-agnostic framework that uses three primary modules to craft a poisoned dataset:
- Target-Group Poisoning: The attacker selects a target group (e.g., dark-skinned patients) and a target class (e.g., "malignant"). They inject a trigger into a subset of this group's training data and change the labels to the target class.
- Non-Target Group Anti-Poisoning: This is the key innovation. To prevent the model from becoming biased against everyone who has the trigger, the attacker also injects the trigger into samples from non-target groups (e.g., light-skinned patients) but keeps their original, correct labels. This teaches the model to ignore the trigger for most people, focusing its "malice" specifically on the target demographic.
- Fairness-Attack Transferable Optimization: Since attackers rarely have access to the victim's specific model architecture, TrojFair uses a surrogate model (a black-box approach). It runs a bi-level optimization process to refine the trigger's shape and mask, maximizing the performance gap between groups while keeping the trigger small and stealthy.
Datasets
The authors evaluated TrojFair on three diverse datasets representing real-world high-stakes scenarios:
- ISIC: Clinical skin lesion images. Sensitive attribute: Age (Target: 0-19 years).
- Fitzpatrick17k: Skin disease images categorized by skin tone. Sensitive attribute: Skin Type (Target: Dark skin).
- FairFace: Facial recognition data. Sensitive attributes: Gender and Age (Target: Female, 0-9 years).
Key Insights and Ideas
- Conditional Discrimination: The most striking insight is that fairness is not a static property. A model can be "fair" 99% of the time but "unfair" exactly when an adversary wants it to be. This renders current "clean-data" fairness audits obsolete against Trojaned models.
- The Power of Anti-Poisoning: Standard Trojan attacks are "group-unaware." By explicitly training the model not to react to the trigger for non-target groups, the authors create a "wedge" of discrimination that is much harder for global detection tools to spot.
- Accuracy vs. Fairness Trade-off: Unlike previous attacks that tanked model accuracy, TrojFair maintains near-perfect performance on clean data. This makes the poisoned model indistinguishable from a benign one to the end-user.
Key Results
- High Effectiveness: TrojFair achieved a Target-group Attack Success Rate (T-ASR) exceeding 88.77% across various models (ResNet, ViT, VGG).
- Minimal Utility Loss: The average accuracy loss on clean data was less than 0.44%, ensuring the model remains useful and unsuspicious.
- Extreme Bias: The "PBias" (bias on poisoned inputs) was significantly higher than clean data bias. For example, on the ISIC dataset, the bias score jumped from ~13% on clean data to over 65% when the trigger was present.
- Resilience to Defense: The authors tested the attack against popular defenses like Neural Cleanse. Standard defenses failed because they look for triggers that affect entire classes; they aren't designed to find triggers that only affect a specific sub-group within a class.
Limitations and Future Work
- Defense Complexity: The authors propose a potential defense by modifying Neural Cleanse to audit every demographic group individually. However, they acknowledge this is computationally expensive and requires the defender to know exactly which sensitive attributes (and combinations thereof) to check.
- Scope: The study focuses primarily on computer vision. Future work could explore how these "fairness Trojans" manifest in Large Language Models (LLMs) or tabular data used in financial lending.
- Trigger Visibility: While the optimized triggers are small, they are still "patches." Exploring more "natural" triggers (like specific lighting conditions or textures) could further increase stealth.
Summary updated