Cross-Channel Audits: Catching Fairwashing by Checking Applicant Recourse Against Regulator Explanations

An institution that uses a black-box model for lending or hiring is increasingly expected to explain it to several audiences at once. The regulator receives a global summary of how the model works. Denied applicants receive counterfactual recourse, a statement of the form "had your income been €4,000 higher, you would have been approved." Prior work shows the regulator-facing summary can be "fairwashed": built to match the black box on most inputs while looking far less discriminatory than it is. The applicant-facing counterfactuals, by contrast, are concrete promises about the real model's behavior, and anyone can check them by acting on them or by resubmitting.

We propose an auditor that pools the recourse statements sent to many applicants and uses them as free, labeled probes of the real model, placed right where fairwashing has to lie: near the decision boundary. The auditor then tests whether the regulator's summary agrees with those probes equally well across demographic groups. The claim is that an institution tailoring explanations to each audience cannot keep all of its stories consistent and also hide its unfairness. Requiring several explanation channels, which is usually seen as a source of new manipulation risk, becomes a way to check one channel against another.

Core Claim

Fairwashing a global surrogate (a simple model shown to the auditor in place of the real one) forces that surrogate to make errors that fall unevenly across groups. Washing The Unwashable : On The (Im)possibility of Fairwashing Detection proves this. Its detector, FRAUD-Detect, compares group-wise confusion matrices between surrogate and black box. It needs black-box labels, though, and it assumes a single explanation channel. A valid counterfactual for a denied applicant $x_i$ is a point $x_i'$ where the real model $f$ flips to approval. So every recourse notice hands the auditor two labeled points, $f(x_i)=0$ and $f(x_i')=1$, sitting on either side of the boundary, without the auditor querying $f$ at all.

We define the cross-channel inconsistency for group $a$ as

$$ D_a = \frac{1}{|C_a|}\sum_{i \in C_a} \tfrac{1}{2}\Big(\mathbb{1}[g(x_i)\neq 0] + \mathbb{1}[g(x_i')\neq 1]\Big), $$

where $g$ is the surrogate disclosed to the auditor and $C_a$ is the set of pooled recourse pairs from group $a$. The audit statistic is the gap between groups in $D_a$, or the group-wise KL divergence of the disagreement patterns, following FRAUD-Detect.

The institution then faces a three-way bind. It can issue counterfactuals that agree with the fairwashed surrogate, in which case they fail when applicants act on them, and the failure is detectable. It can issue counterfactuals that are valid for $f$, in which case they reveal $g$'s group-skewed errors at the boundary. Or it can give up on fairwashing.

The hypothesis is that boundary-located probes detect fairwashing with far fewer samples than random labeled queries. This is the regime where Auditing Local Explanations is Hard shows random black-box auditing becomes hopeless: its sample needs grow as one over locality, and locality shrinks exponentially with dimension.

Positioning

Three papers supply the threat. Fairwashing: the risk of rationalization builds LaundryML, a rule-list search that reaches 90.8% fidelity at 0.058 unfairness on Adult Income. It also rationalizes individual outcomes, but it treats the global and local attacks as separate, static deceptions. Characterizing the risk of fairwashing shows that fairwashed surrogates generalize to held-out data with a fidelity drop under 1–2% and transfer across architectures. Its FaiRS measure is a diagnostic of how much room for this exists, not a defense. FRAUD-Detect supplies our statistic, and our method reuses it. We change where the labels come from and whether the adversary must stay consistent across channels.

Two papers motivate the multi-audience setting itself. Adaptive Explanations via Direct Preference Optimization tailors explanations to each listener, raising listener accuracy from 84.62% to 98.75% on CheXpert, but it assumes a cooperative speaker. EXAGREE searches the set of near-equally-accurate models for stakeholder-aligned explanations under an honest deployer. We take their tailoring machinery and hand it to a dishonest deployer.

The structural idea comes from security. Securing Data against Limited-Knowledge Adversaries in Distributed Storage Systems catches attackers who look valid locally by checking global correlations they cannot see. Here, each applicant sees only their own notice, but the pooled notices reveal a global inconsistency.

One paper constrains us. On Counterfactual Metrics for Social Welfare shows that giving a regulator partial extra information can increase gaming. We therefore test whether adding a channel can ever make detection worse, rather than assuming it helps.

Research Program

We first build a multi-channel adversary. The institution trains a black box $f$, then jointly optimizes three things. The first is a disclosed surrogate $g$, either a logistic model or a rule list, with high fidelity to $f$ and a low demographic parity gap. The second is a counterfactual generator for denied applicants. The third is a penalty on the expected inconsistency the institution believes the auditor can measure. Varying the penalty weight traces a frontier between surrogate unfairness and detectability. We include variants in which the attacker knows the auditor's statistic and in which it does not, and variants that issue different-looking counterfactuals per group to dilute pooling.

Second, we build the auditor around $D_a$. We add a recourse-cost disparity check, since one evasion is to give valid but much more expensive recourse to one group, and that cost gap is itself evidence. We also add an option to spend a small budget of real black-box queries, so the auditor can trade pooled notices against queries.

Third, we analyze the bind. We extend the FRAUD-Detect argument to show that any surrogate whose parity gap is below the black box's by $\delta$ must misclassify a group-skewed fraction of points near the boundary, scaled by $\delta$. Valid counterfactuals sample exactly that region. This would explain why pooled recourse avoids the dimension blow-up that random sampling suffers.

Counterfactual generation is NP-hard for tree ensembles and ReLU networks (On the Hardness of Computing Counterfactual and Semifactual Explanations in XAI). This limits the institution, which must use heuristics. It does not limit the auditor, which only checks counterfactuals it is given.

Baselines

The naive reference point is the audit regulators perform today. They take the disclosed surrogate at face value and measure its demographic parity on a held-out test set. Characterizing the risk of fairwashing already shows this is fooled: surrogates appear 50% less unfair at over 90% fidelity. We expect near-chance detection, and we report it to anchor the scale.

The strongest existing method is FRAUD-Detect, given the same number of black-box labels as there are pooled recourse pairs, drawn uniformly from the test distribution. It runs on the same datasets (COMPAS, Adult Income, Bank Marketing, plus German Credit and Default Credit) and the same black boxes (DNN, AdaBoost, random forest), and it gets the same tuning budget for its decision threshold. Its paper reports that fairwashed surrogates show group KL divergences at least 4× higher than honest ones, a gap above 0.6 on COMPAS. It also reports that an informed adversary evading it must accept a parity gap above 10%. Beating it means two things. First, at matched label budget, our audit should reach the same detection power (AUC between honest and fairwashed surrogates) with substantially fewer labels. Second, against the adaptive attacker, the parity gap the attacker must concede to evade our audit should be at least FRAUD-Detect's.

We also report FaiRS as a non-interactive diagnostic. It tells us how much fairwashing room the Rashomon set offers each dataset, so detection results can be read relative to the available slack.

LaundryML serves as the reference static attacker. Its Adult Income operating point is 90.8% fidelity at 0.058 unfairness.

Two ablations isolate the key ingredient. The first replaces recourse pairs with random labeled points at matched count. This separates "labels for free" from "labels at the boundary." The second runs the consistency check without splitting by group. This separates boundary information from the group-disparity argument.

Why not just…?

Why not just run FRAUD-Detect with black-box queries? This is the real threat to the proposal. If auditors already have generous query access and enough budget, cross-channel pooling adds convenience but not power. Our matched-budget comparison settles it. If uniform random labels match pooled recourse at equal count, the contribution shrinks to "recourse notices are a legal source of labels," which still matters in regimes without query access but is a weaker claim. We will report it plainly if so. The sample-complexity result in Auditing Local Explanations is Hard suggests random queries degrade with dimension in a way boundary probes may not. That remains a prediction to test.

Why not just mandate inherently interpretable models? If the deployed model really is the interpretable one, fairwashing disappears, and that is the right policy wherever it costs little accuracy. The file contains no reported accuracy gap on these benchmarks. Mandates also only cover what is disclosed. LaundryML's deceptive surrogate is itself a certifiably optimal rule list, so being interpretable does not guarantee that a model is the one deciding. Our audit tests exactly that link between disclosed and deployed.

Why not just require one standardized explanation format, such as SHAP-based reason codes? A single channel removes the cross-check rather than the incentive to deceive. Outcome rationalization in Fairwashing: the risk of rationalization shows that per-person explanations can themselves be laundered. A single format also fails the audiences it does not suit (Requirements of eXplainable AI in Algorithmic Hiring). We test this alternative by running the adaptive attacker with only reason codes disclosed and comparing achievable concealment.

Why not just verify each applicant's counterfactual on its own? Checking that recourse is valid catches a lying counterfactual. But a fully honest counterfactual paired with a fairwashed global surrogate passes every individual check. The deception lives only in the relation between channels, which the no-group-split and group-split ablations measure directly.

Proposed Experiments

The first decisive check is the attacker's frontier. For each dataset and black box, we plot the best surrogate parity gap achievable against our audit's detection rate as the consistency penalty grows. The claim succeeds if, at a fixed false-alarm rate, evading detection forces the surrogate's parity gap most of the way back toward the black box's. It fails if attackers retain most of the 50% concealment while passing the audit.

The second check is sample efficiency. We plot detection AUC against the number of pooled pairs versus random labels. We also increase dimension using synthetic Gaussian data with known ground truth, in the style of OpenXAI's SynthGauss (OpenXAI), to test whether the gap widens with dimension as predicted.

The third check targets the backfire risk raised by the social-welfare metrics paper. We add a third channel, reason codes, and test whether the attacker can use it to mislead the auditor's pooling so that detection power falls below the two-channel audit.

A final robustness check models partial pooling. We vary the fraction of denied applicants who share their notices, from 1% to 50%, and let the attacker know which groups share less. This gives the participation level at which the audit stops working, which is the number a regulator would actually need.