Most methods that make text classifiers "gender-fair" reduce gender to a single bit. A model that reads a biography and predicts an occupation is trained so that its internal representation carries no information about whether the subject is labelled male or female. That one bit is all such methods can use, because it is the only gender annotation available at scale. The bit is wrong for exactly the people who most need protection. A study of 398 self-identified non-binary French writers found no single "non-binary signature". Some wrote consistently with masculine grammatical forms, some consistently with feminine ones, and some mixed forms or used new ones such as the point médian ("heureux·se"). Only 95 of 1,564 gendered expressions used explicit non-binary forms (Characterizing non-binary French). The useful signal is both rare and structured.
Molecular and materials modeling has faced this kind of scarcity for years. There the rich object, 3D atomic structure, is expensive and available for few examples, and the cheap object, a sequence or chemical formula, is everywhere. The working answer was neither to give up on structure nor to wait for more of it. A structure-aware teacher model is trained where structure exists, and its knowledge is passed to a structure-free student that runs everywhere. This note proposes the same move for gender. We treat the gender-marking structure of a text as the scarce privileged object: which words grammatically mark the person being described, and whether each mark is masculine, feminine, neutralized or neomorphic. A teacher sees this structure; a text-only student learns to reproduce the teacher's continuous "gender-expression profile". The profile then replaces the binary label as the sensitive variable in a fairness penalty, so a downstream classifier can be made independent of gender expression without ever forcing a text into one of two bins.
Core Claim
The claim has two parts, and the experiments test them separately. First, a fairness penalty defined against a continuous, multi-dimensional profile of gender marking protects non-binary and mixed-marking texts better than the same penalty defined against a binary or three-class label. The profile is a point on a simplex over the four marking types, plus a count of how many marks the text contains. Second, when explicit marking annotations exist for only a small subset of texts, distilling a structure-aware teacher into a text-only student yields better profiles on rare and unseen marking forms than either training a text classifier directly on the scarce labels or applying hand-written extraction rules. Better profiles in turn yield better fairness. The first part could hold while the second fails. In that case the contribution reduces to the profile itself, and we say so in advance.
The penalty is the Wasserstein dependency measure used by Fair Text Classification via Transferable Representations. That work pushes text representations toward independence from a sensitive attribute, and it infers the attribute in unlabelled domains with a secondary "demonic" model. It restricted gender to a binary attribute and named continuous or non-binary attributes as unsolved, because thresholds for binning were unclear. Our objective is
$$ \min_{\theta}\; \mathbb{E}\big[\ell(f_\theta(x), y)\big] \;+\; \lambda\, W\!\big(P_{h,s},\; P_h \otimes P_s\big), $$
where $x$ is a text, $y$ its task label, $f_\theta$ the classifier with internal representation $h = h_\theta(x)$, and $s = g(x)$ the gender-expression profile produced by the distilled student $g$. The term $W(P_{h,s}, P_h \otimes P_s)$ is the Wasserstein distance between the joint distribution of representation and profile and the product of their separate distributions, estimated with a learned critic. Because this distance is defined for continuous $s$, no binning is needed. That is the precise sense in which we answer the open problem WFC left. Texts with no gendered marks receive a flagged profile and are excluded from the penalty, so the method never invents a gender expression the text does not contain.
Why the structural-scarcity lens, and what it borrows
Two papers on structural scarcity supply the method. DISTAL distills internal embeddings from ALIGNN, a crystal-structure-aware teacher, into a student that sees only the chemical formula. It pretrains on 100,000 unlabelled virtual compositions and improves on the reference benchmark in 37 of 39 materials tasks. Its ablations show that the distilled structural features beat raw composition descriptors alone. Transformer-M trains one network on 2D-only, 3D-only and joint molecular data through a switch that turns structural channels on or off, so the network still works when 3D input is absent. We reuse DISTAL's teacher-to-student distillation on a large unlabelled corpus. We reuse Transformer-M's mixed-availability training, because our supervision comes in three grades: some texts have full marking annotations, many have only binary labels, and most have nothing.
DISTAL also constrains the design. It failed on magnetic moments and piezoelectric constants, properties that depend on exact symmetry the formula cannot recover. The analogue here is neomorphic forms, which are rare and orthographically varied, and they are the place we expect distillation to be weakest. The experiments report them separately rather than averaging them away.
The gender-bias literature motivates the profile itself. ADELE debiases BERT with adapters trained by counterfactual swapping of gendered terms, and it is explicitly restricted to binary gender. The authors note that term-swapping cannot produce gender-neutral syntax without breaking fluency. GECOBench builds masculine, feminine and non-binary variants of the same sentences, but by rule-based pronoun replacement that its authors call simplified. GenWriter rewrites biographies by retrieving the opposite-gender case, which has no meaning for a non-binary subject. All three treat the non-binary case as a third value or an absence. The French corpus shows it is neither. A non-binary author's text may be marked exactly like a man's, and a classifier that sees "binary-masculine" there has learned nothing false about the text, only about its author. This is why the profile describes marking in the text and not the identity of the author. That keeps the method from becoming a non-binary-identity detector, a use we consider harmful and do not pursue.
Research Program
The first component is the marking-structure annotation that the teacher consumes. For French, we extend the self-referential "je + être" analysis of the non-binary French corpus to all words that agree in gender with a person referent: determiners, adjectives, participles and nouns. Each mark is tagged as masculine, feminine, neutralized or neomorphic. The 1,564 annotated expressions seed the set, and a rule-based extractor proposes candidate marks that annotators verify. For English we use the GECO subject and all-words variants, where the manipulated tokens are known by construction, together with pronoun and gendered-noun marks in BiasBios biographies. The result is a small, gold-structured set in each language.
The second component is the teacher. It is a text encoder whose input is augmented with the marking graph: each token is tied to its referent, and each edge carries the agreement type. The teacher is trained to predict the profile and the downstream labels. The third component is the student, a plain text encoder. It is trained with a Transformer-M-style switch over three supervision regimes: texts with full structure, where it matches the teacher's embedding and the gold profile; texts with only a binary label, where it predicts a coarse masculine-versus-feminine marginal; and unlabelled texts, where it matches the teacher's embedding on teacher-parsed inputs. The unlabelled pool is the remainder of the 18,662-text French corpus and of BiasBios. The fourth component is the fair downstream classifier, trained with the objective above, with the student's profile as $s$.
Baselines
The naive reference point is a fine-tuned BERT-family occupation classifier with no fairness intervention, trained on BiasBios. It sets the accuracy ceiling and the unmitigated gap on binary and non-binary variants.
The strongest existing method for the core question is WFC with its demonic model, trained on binary BiasBios gender labels exactly as published (WFC). It is the closest competitor because it uses the same dependency penalty, so any gain we show must come from the sensitive variable rather than the regularizer. No exact WFC score was found on Lacuna for BiasBios; the file reports only that it ranks among the top methods on its accuracy-fairness measure. We will therefore rerun it rather than quote it.
On binary BiasBios, the strongest reported numbers come from GenWriter. It reduces the true-positive-rate gap for nurse biographies from -0.09 to -0.01 and for surgeon biographies from 0.08 to 0.03, at 89.15% average accuracy against 89.55% on original text (GenWriter). We must match these gaps within noise on the binary split, at no worse accuracy. On the extrinsic side, ADELE reports a reduction in BEC-Pro bias from 1.33 to 0.39 with 84.13% MNLI accuracy (ADELE). We include ADELE as a counterfactual-augmentation baseline, extended with GECO-style rules that substitute "they".
Three ablations isolate the key ingredients. The first, a three-class WFC, replaces the binary label with masculine, feminine and non-binary labels taken from GECO variants and the French annotations; it isolates continuity versus categories. The second, a rule-profile WFC, computes $s$ directly with the hand-written extractor and no learned student; it isolates distillation. The third, a direct student, trains on the scarce gold profiles without teacher matching or the unlabelled pool; it isolates the teacher.
Every variant uses the same backbone, the same training data, and the same number of hyperparameter trials, selected on a shared validation split. Selection uses a fixed sum of accuracy and binary-gap criteria, so that no method is tuned on the non-binary test sets. Beating the field means the following. The full method must have a lower prediction-shift and gap on non-binary and mixed-marking test texts than WFC and three-class WFC, with confidence intervals over five seeds that do not overlap. It must also lose at most one point of accuracy relative to the best baseline.
Why not just…?
Why not just add a third "non-binary" class to WFC? This is the most natural extension, and it is ablation one. The French evidence predicts it will fail in a specific way. Most non-binary authors mark consistently masculine or feminine, so a categorical attribute either mislabels them or must be inferred from rare neomorphs. With 95 neomorphic expressions from 24 people, that third class is too sparse to estimate dependence against. The experiment will show whether three-class WFC already closes the gap on the mixed-marking test slice. If it does, the continuous profile is unnecessary and we will report that.
Why not just extend counterfactual swapping to "they" and neopronouns? ADELE's authors state that term-swapping cannot produce valid gender-neutral syntax. The French data show that real non-binary text follows no single substitution rule. Swapping also only reaches lexical marks, while a dependency penalty also removes correlated stylistic proxies. We run the extended swapping baseline anyway. Its expected failure appears on the naturalistic French slice, not on GECO, where its rules match the test-set construction, and this asymmetry is itself a check.
Why not just rewrite texts to be gender-neutral before classifying? This is a real threat when the deployment allows rewriting. GenWriter's case-based rewriting beats LLM-only rewriting. The LLM-only version reduced the gap by only 44.4% for nurses and 12.5% for surgeons, cut accuracy to 85.11%, and inserted new subtle gendered words. But GenWriter relies on opposite-gender case retrieval, which has no non-binary analogue. We compare against LLM-only neutral rewriting and against GenWriter on the binary split. If rewriting matches our gaps on the non-binary slices at equal accuracy, the representation-level approach is not justified for that setting.
Why not just use the rule-based extractor and skip the teacher? If the rules are reliable, distillation adds nothing. That is why the rule-profile WFC is a first-class ablation and not a straw man. We expect the rules to break on orthographic variants of neomorphs and on cross-domain text, and the held-out-form test below is designed to show where they break.
Proposed Experiments
We evaluate on four test settings. The first is binary BiasBios, using the per-occupation true-positive-rate gap and accuracy, to confirm no regression against GenWriter's reported figures. The second is GECO, where for every sentence triple we measure how far the classifier's prediction and representation move across the masculine, feminine and non-binary variants. Lower movement is better, and we report the subject-only and all-words corpora separately. The third is the French non-binary corpus, split by the three marking clusters. The downstream task is predicting the source platform or topic, a label unrelated to gender. We measure whether accuracy and calibration differ across clusters and across matched texts by binary-identified authors. The fourth is profile quality itself: mean error of the student's profile against gold annotations, reported separately for masculine, feminine, neutralized and neomorphic marks.
Three checks are decisive. The first is held-out forms. We remove one family of neomorphic forms entirely from training (for example, all point-médian forms) and test whether the distilled student assigns those texts neomorphic mass, where the direct student and the rules do not. This is our analogue of DISTAL's symmetry-sensitive failures, and it is the test most likely to sink the distillation claim. The second is the mixed-marking slice. If the continuous profile shows no advantage over three-class WFC on texts that combine masculine and feminine marks, the first half of the core claim fails. The third is the label budget. We vary the amount of gold structure from the full annotated set down to a tenth of it, and check that the full method degrades more slowly than the direct student. If distillation helps, the benefit should appear at the low end, where scarcity bites.
References
- Characterizing non-binary French: A first step towards debiasing gender inference
- Fair Text Classification via Transferable Representations
- DISTAL: Distillation and Self-Supervised Pretraining for Structure-Agnostic Materials Property Prediction
- One Transformer Can Understand Both 2D & 3D Molecular Data
- Sustainable Modular Debiasing of Language Models
- GECOBench: a gender-controlled text dataset and benchmark for quantifying biases in explanations
- GenWriter: Reducing Gender Cues in Biographies through Text Rewriting