When text classifiers are trained by federated learning, many clients such as hospitals, phones or offices each train a copy of the model on their own private text. A server then averages their updates, and the raw text never leaves its owner. The averaged model is often much worse than one trained on pooled data. On 20Newsgroups with skewed client data, the best federated optimizer reaches about 55% accuracy, against about 86% for centralized training (FedNLP). The usual explanation is client drift: each client pulls the shared weights toward its own narrow data, and the pulls partly cancel when averaged.
This note proposes a simple, testable remedy for one specific kind of drift. Each input's learned embedding would be concatenated with a short vector of deterministic linguistic statistics: dependency distances, clause ratios, readability scores and vocabulary richness. These statistics come from a fixed parser and fixed formulas, so they have no trainable parameters and cannot drift. If every client standardizes them with the same global statistics, all clients describe their text in identical coordinates. The claim is not that linguistic features fix federated learning in general. The claim is narrower: they should shrink the federated-to-centralized gap when clients differ in how they write (register, noise level, style), and should do little when clients differ only in which labels they hold. The experiments are designed to separate these two cases.
Core Claim
Consider a frozen transformer with small trainable adapters, which is the realistic setting for edge devices. Add a globally normalized syntactic feature vector, fused at the classifier input. The prediction is that this reduces the dispersion of client updates and the accuracy gap to centralized training under feature-shift partitions, but not under pure label-skew partitions. The benefit should come mainly from syntactic rather than lexical features. It should disappear if the features are replaced by an equal number of learned or random dimensions.
The last point matters most. It is what distinguishes "structural anchoring" from "a few extra inputs." The syntactic-over-lexical prediction follows directly from Lexical Features Are More Vulnerable, Syntactic Features Have More Predictive Power. That paper found lexical statistics shift sharply under word-deletion noise while syntactic statistics stay stable and carry roughly 2× (clinical transcripts) to 7× (IMDB) more regression importance. If clients differ in noise level, lexical features become a client identifier rather than a shared signal, which should add drift.
Relation to Prior Work
Three lines of work supply the ingredients.
Centralized fusion. Hybrid fusion methods concatenate symbolic features with dense embeddings and train centrally:
- OSTAR fuses dependency-distance, readability and coherence statistics with RoBERTa's sentence embedding. It reports 94.04% average clean F1 and 81.27% F1 under perturbation on DeepFake.
- Disambiguating Spatial Prepositions Using Deep Convolutional Networks fuses PoS and lexical-database indicators into a convolutional network.
- katildakat at SemEval-2021 Task 1 adds handcrafted features to contextual embeddings for complexity prediction.
This proposal reuses OSTAR's feature families and its simple concatenation-at-the-sentence-embedding design. It does not reuse OSTAR's contrastive objective, which would confound the question. None of these works asks what fused features do to parameter averaging across heterogeneous clients.
Federated NLP. This line supplies the setting and the baselines, and motivates the gap:
- FedNLP provides partition protocols: Dirichlet label skew, and Sentence-BERT clustering for feature shift. It also provides the FedAvg, FedProx and FedOPT results and the 55% vs 86% gap.
- AutoFedNLP shows that adapter-based federated fine-tuning (FedAdapter) converges up to 155.5× faster than full fine-tuning and cuts traffic from 2.1 TB to 9.9 GB. This is why the proposal works with adapters rather than full weights.
- A Systematic Evaluation of Federated Learning on Biomedical NLP constrains the design. It finds FedProx underperforms FedAvg unless its penalty weight is tuned carefully, so any drift remedy must be compared with a properly tuned FedProx.
Nearest connections. Two papers come closest to joining the directions:
- FedLFC uses linguistic knowledge only as fixed, a priori language-family labels for grouping clients' LoRA adapters. It reaches 88.7% on XGLUE news classification against 85.3% for FedAvg.
- Multilingual Federated Prompt Tuning measures "linguistic distance" as a cosine similarity between dense embeddings.
This proposal differs from both in using per-example linguistic measurements as model inputs, not metadata about clients or properties of embedding space.
One citation constrains the proposal from the other side. MorphingMinds at SemEval-2024 Task 10 highlights that feature extraction degrades on code-mixed and informal text. A client with noisy text may therefore receive worse features, not stable ones. The noise-heterogeneous partition described below tests this directly rather than assuming it away.
Research Program
Model. The model is DistilBERT with adapters, following the FedAdapter recipe. The classifier input is the pooled sentence embedding concatenated with a feature vector of roughly 20 to 40 dimensions, which is small next to the 768-dimensional embedding. The features are:
- mean and maximum dependency distance
- clause and subordinate-clause ratios
- sentence length
- parse depth
- Flesch-Kincaid grade
- type-token ratio and word-frequency statistics, as the lexical group
- embedding variance across sentences, as a coherence proxy
Each client computes the features once per example with the same off-the-shelf parser.
Normalization. The one design choice specific to federated learning is how to standardize these features. Standardizing locally per client would silently re-introduce client-specific coordinates and undo the anchoring. Instead, clients send only per-feature counts, sums and sums of squares to the server under secure aggregation, and receive back a single global mean and variance. This is a few dozen numbers per client, sent once, and reveals far less than a gradient.
Partitions. The model is trained with FedAvg, FedProx and FedOPT over three partition families:
- FedNLP's Dirichlet label-skew splits.
- FedNLP's Sentence-BERT feature-shift clusters.
- A new noise-heterogeneous partition. Each client receives a different word-deletion rate (0%, 20%, 40%, 60%), following the Novikova et al. protocol, to mimic sites with different transcription quality.
Measurements. Beyond accuracy, the analysis measures three things:
- per-round dispersion of client adapter updates, as the mean cosine distance to the averaged update;
- rounds to reach a fixed accuracy;
- worst-client accuracy.
Baselines
The do-nothing reference is FedAvg with adapters and no linguistic features, on identical partitions, seeds and round budgets. Local-only training and centralized pooled training bracket it from below and above. For 20Newsgroups the reported anchors are about 86% centralized and about 55% for the best federated optimizer under non-IID skew (FedNLP). These were full-model results. The adapter-only versions will be re-run under our settings, not assumed.
The strongest existing method is FedOPT (server-side adaptive optimization), plus FedProx with its penalty weight tuned per partition. FedOPT is the best optimizer in FedNLP. FedProx is the direct competitor on drift, and the biomedical evaluation shows it must be tuned to be fair. Each optimizer receives the same hyperparameter-search budget with and without linguistic features. The features are tested as an addition to each optimizer, not only against FedAvg. Beating this baseline means the hybrid variant closes a statistically meaningful share of the centralized gap over five seeds on feature-shift and noise partitions, beyond what tuned FedOPT or FedProx achieve alone.
For the multilingual check, the reported baseline is FedLFC at 88.7% accuracy on XGLUE news classification (FedLFC). Centralized training there is only 88.1%, so there is essentially no gap left to close. XGLUE is therefore used only to test whether multilingual feature extraction harms accuracy, not as the main evidence.
Two ablations isolate the key ingredient:
- A capacity control replaces the linguistic vector with an equal-dimensional vector of learned features. This is a second pooled projection of the backbone, trained like the adapters and so subject to drift. A fixed random projection of the embedding is also tested.
- A normalization control standardizes features locally per client instead of globally.
A further ablation compares the syntactic group alone with the lexical group alone.
Why not just…?
Why not just use a better federated optimizer? FedOPT is the best optimizer in FedNLP and still leaves roughly a 31-point gap on non-IID 20Newsgroups. FedProx is fragile to its penalty weight, per the biomedical evaluation. These methods act on the loss rather than the input, so they are complementary, and every optimizer is run with and without the features. If tuned FedOPT alone matches FedOPT plus features on the feature-shift partitions, the proposal fails on its main claim, and the experiment is set up to show that.
Why not just use adapters or LoRA without features? Parameter-efficient tuning cuts communication, as AutoFedNLP and FedLFC show. FedLFC still needed external client grouping to avoid interference, which suggests adapters alone do not remove heterogeneity. The adapter-only model is the do-nothing reference, so any gain is measured directly on top of it.
Why not just add more input dimensions of any kind? This is the most serious threat. A few extra inputs might help simply by adding capacity or giving the classifier a shortcut. The learned-projection and random-projection controls address it directly. If either matches the linguistic features, the "drift-free anchor" explanation is wrong and should be abandoned.
Why not just personalize the classifier head per client? Under label skew, personalization is probably better, and the proposal predicts no advantage there. The question here is the quality of a single global model that must serve unseen clients. Personalization is reported for completeness, and the label-skew partition doubles as a negative control.
Proposed Experiments and Decisive Checks
The primary benchmark is 20Newsgroups under FedNLP's three partition families plus the noise partition, with 100 clients and 10 sampled per round. Two secondary checks follow:
- The same noise-heterogeneous protocol on IMDB, where Novikova et al. found the largest syntactic-to-lexical importance ratio.
- XGLUE news classification, to test whether extractor quality differs across languages. This matters because parsers for some languages are weaker, which could turn the anchor into noise.
Per-client parse time on a CPU is also reported, because extraction runs on the edge device.
The result pattern that supports the claim has five parts:
- The hybrid model gains on the feature-shift and noise partitions and not on label skew.
- The dispersion of client updates falls in the same partitions where accuracy rises.
- The learned-projection and random-projection controls do not reproduce the gain.
- Global normalization beats local normalization.
- Syntactic features help, while lexical features help less or hurt on the noise partition.
Any one of these failing is informative. Gains without lower dispersion would mean the features help for a reason other than drift. Gains under label skew would contradict the mechanism. Equal gains from random dimensions would make it a capacity effect.
References
- FedNLP: Benchmarking Federated Learning Methods for Natural Language Processing Tasks
- Lexical Features Are More Vulnerable, Syntactic Features Have More Predictive Power
- OSTAR: Optimized Statistical Text-classifier with Adversarial Resistance
- Disambiguating Spatial Prepositions Using Deep Convolutional Networks
- katildakat at SemEval-2021 Task 1: Lexical Complexity Prediction of Single Words and Multi-Word Expressions in English
- AutoFedNLP: An efficient FedNLP framework
- A Systematic Evaluation of Federated Learning on Biomedical Natural Language Processing
- FedLFC: Towards Efficient Federated Multilingual Modeling with LoRA-based Language Family Clustering
- Breaking Physical and Linguistic Borders: Multilingual Federated Prompt Tuning for Low-Resource Languages
- MorphingMinds at SemEval-2024 Task 10: Emotion Recognition in Conversation in Hindi-English Code-Mixed Conversations