Hospitals, labs and banks often hold the same kinds of measurements for different people: the same gene panel, the same MRI voxels, the same lab tests. They want to agree on a small, interpretable set of variables that predicts an outcome, without pooling raw records. Centralized practice handles very high-dimensional data with a hybrid funnel. A cheap statistical filter first cuts thousands of features down to a few hundred. An expensive multivariate search, called a wrapper, then picks the final subset. Moving this funnel into federated learning (training a shared model by exchanging only parameter updates, never raw data) runs into two problems. The wrapper needs hundreds of full training runs, and each federated run costs many rounds of communication. The filter scores features one at a time, so it permanently discards features that are useless alone but informative together.
We propose a funnel for horizontal federated learning, the setting where clients share a feature space but hold different samples, that addresses both problems. The wrapper is replaced by a single federated training run of a neural network with learnable on/off gates on each input. The filter is computed exactly from securely summed per-class statistics. Its scores act as soft priors on the gates rather than hard cutoffs. A small, rotating reserve of filter-rejected features is fed through the gates on every round, so a feature the filter wrongly dropped can still be picked. The claim is that this recovers interaction-only features under uneven client data at roughly the communication cost of one ordinary federated model. Doing the same with a federated genetic-algorithm wrapper would cost one to two orders of magnitude more.
What already exists and where this sits
Federated feature selection has been studied mostly in the vertical setting, where parties hold different columns for the same people. LESS-VFL pre-trains a rough joint model, runs group lasso on the server over party embeddings, and sends the selected components back as local proxy targets. Vertical Federated Feature Screening runs a coarse-to-fine marginal distance-covariance screen over more than 100,000 features. The MSEs credit risk model is the only explicit filter-then-wrapper pipeline in the file. It filters locally inside each silo (mRMR, PCA) and then wraps with encrypted boosted trees on 170 records. These papers show that funnels pay off when parties are federated. Their mechanisms, however, depend on column-partitioned data and do not transfer to our setting.
In the horizontal setting, three prior methods constrain the design.
- SeiFS builds a cryptographically secure filter-ranking and top-k protocol. It has no multivariate stage.
- FedGCUR shows that column-pivoted QR can be computed exactly from securely aggregated inner products. Its selection ignores the labels.
- FedCMFS learns causal Markov blankets with distributed conditional-independence tests. Its authors report that these tests weaken on small or noisy clients.
We take two things from this line of work. FedGCUR supplies the idea that aggregated sufficient statistics can reproduce a centralized computation exactly. SeiFS supplies the warning that heavy cryptography is expensive. Our filter therefore needs only additive secure aggregation, and every multivariate decision is made by training a model rather than by comparing statistics.
The method itself comes from Stochastic Gates (STG). STG learns noisy gates jointly with network weights and recovered the 5 true MADELON features where LASSO, random forests and XGBoost did not. The centralized funnels supply the prior idea. MF-GE uses aggregated filter scores to bias where a genetic search starts, and CGAFS uses a Fisher-score sieve before a genetic algorithm. Both still pay for a population of wrapper evaluations. We keep the score-as-prior idea and drop the population. No paper in the file runs stochastic gates under federation, and none studies how gate parameters behave when averaged across clients with different data.
Core Claim
In horizontal federated learning on data with many more features than samples, a three-part funnel matches a federated genetic-algorithm wrapper on downstream accuracy and true-feature recovery while using at least 10x less communication. The three parts are an exact secure-sum filter, a gated network over the filtered pool trained with FedAvg (averaging client weight updates), and a rotating reserve of rejected features. The reserve specifically recovers features with near-zero marginal signal, which a hard-thresholded filter-plus-gates pipeline loses.
Research Program
Stage 1: the filter. Each client computes per-class counts, sums and sums of squares for every feature, plus binned counts for mutual information. The clients submit these through secure aggregation. Because these statistics add up across clients, the server obtains exactly the centralized ANOVA F-value, Fisher score and binned information gain without seeing any single client's values. An optional arm adds Gaussian noise to the sums to give a differential-privacy guarantee, and we measure how ranking quality degrades as the privacy budget tightens. This addresses Gap 4 of the file at the cost of one round of communication.
Stage 2: gated training. The pool contains the top $k$ features plus a reserve of $r$ features drawn at random from those the filter rejected. A network with a gate on each pooled input is trained by FedAvg over gate means and weights. The local objective adds a per-feature gate penalty whose strength depends on the filter score:
$$ \mathcal{L}_c = \hat{\mathbb{E}}_{c}\big[\ell(f_\theta(x \odot z), y)\big] + \lambda \sum_{d} \frac{\Phi(\mu_d/\sigma)}{1 + \alpha s_d}. $$
Here $\hat{\mathbb{E}}_c$ is the average over client $c$'s data and $\ell$ is the prediction loss. The network $f_\theta$ has weights $\theta$, and $z$ is the vector of stochastic gates with means $\mu_d$ and noise scale $\sigma$. $\Phi$ is the standard normal CDF, so $\Phi(\mu_d/\sigma)$ is the probability that gate $d$ is open, as in STG. The normalized filter score is $s_d$, and $\alpha$ sets how much the filter is trusted. A feature with a strong filter score is cheaper to keep but is never forced in. A reserve feature starts with $s_d$ near zero and must earn its gate through the prediction loss.
The reserve. Every $T$ rounds, reserve features whose gates stayed closed are swapped for fresh draws, and any reserve feature whose gate opened is promoted into the core pool. The reserve extends STG's "second chance" from individual features to the filter boundary itself. With a fixed communication budget it is the only way a purely interactive feature can enter the model.
Analysis. We also log the spread of client gate updates for each feature. This tests the concern raised in Gap 3, that clients with different data push the same gate in opposite directions. If the spread is large, plain averaging is insufficient, and we add a variant that keeps disputed gates open as possible site-specific signal.
Baselines
Naive reference. Each client runs LASSO or random-forest importance locally, and the server takes a majority vote over the selected sets before training a global model. This is the obvious simple approach. LESS-VFL found the analogous "Local Lasso" baseline clearly worse in the vertical setting, but nobody has measured it in the horizontal setting, so we treat it as a live competitor. A second reference trains on all features with no selection, which bounds the accuracy cost of sparsity.
Strongest existing method. The strongest competitor is a federated hybrid funnel built from the best centralized ingredients. It applies the same exact secure-sum Fisher filter and then runs a genetic-algorithm wrapper in the style of CGAFS and the distributed GA. Each candidate subset is scored by a short federated training run. This is the "obvious idea" at the intersection, and it is the method most likely to match our accuracy. It gets the same filter, the same model class and the same total tuning budget. We report it at its natural communication cost and also capped at our method's cost.
The file gives centralized reference points:
- the distributed GA raised MLP F1 on Gina from 0.875 to 0.889;
- CGAFS reduced Colon to 12.98 features on average;
- centralized STG reached a concordance index of 0.633 on METABRIC with 2 features.
No reported federated result for a GA wrapper was found on Lacuna. For filter-only horizontal methods, the file contains no reported downstream accuracy for SeiFS (only runtime) and none for FedGCUR (only reconstruction error). We therefore implement the filter-only funnel, the label-blind FedCPQR selection, and FedCMFS-style conditional-independence selection ourselves, using their published procedures.
Ablations. Four ablations isolate each ingredient: our method without the reserve (hard top-$k$ cut); our method with $\alpha = 0$ (no filter prior); federated STG on all features with no filter; and centralized STG on the pooled data as an upper bound.
What counts as beating the strongest baseline. On MADELON and the interaction-only synthetic data, the GA funnel must not exceed our true-feature recovery F1 or AUC by more than one standard error across seeds, while our method uses at most one tenth of its communicated bytes. On both, our method must also beat the no-reserve ablation on recovery of interaction-only features.
All methods use identical client splits, the same evaluation sets, the same network architecture where applicable, and the same number of hyperparameter trials. Communication is measured in bytes rather than rounds.
Why not just…?
Why not just select locally and vote? With label-skewed or very small clients, a feature that is informative overall can look useless at every single site. Small-cohort studies such as the aneurysm biomarker screen already show unstable selections at 35–40 patients. Still, if clients are close to identically distributed, voting may be good enough. Our Dirichlet label-skew sweep shows at what level of client heterogeneity voting breaks down, and we will report honestly if it never does on the real multi-site data.
Why not just add L1 to FedAvg? Averaging client weight vectors whose zero entries fall in different places produces dense global weights. L1 also misses interactions, as STG's MADELON comparison shows. The federated-L1 baseline directly measures both effects: how many features survive in the global model and how much recovery F1 it loses on MADELON.
Why not just run federated stochastic gates on all features? This is the real threat to the funnel part of the proposal. When $p$ is in the hundreds, as in MADELON (500), Gina (451) or METABRIC (221), it may simply win, and the filter would add nothing. Our argument rests on the first-layer cost of $p$ times the hidden width per round, which dominates at microarray and MRI scale. The no-filter ablation run at matched bytes settles this question. If it matches our method at every scale, we will drop the filter and report the paper as "federated gates are enough."
Why not just run the federated GA, since it is the obvious hybrid? It probably works. Its cost is what we expect to rule it out: hundreds of federated training runs. The capped-budget comparison shows whether it degrades below our method when given the same bytes.
Proposed Experiments
The benchmarks are MADELON and the Agnostic Learning Challenge sets Gina, Sylva and Hiva, split across 5 to 20 clients under Dirichlet label skew. We add synthetic XOR-style data whose informative features have exactly zero marginal correlation with the label, so that true-feature recovery can be scored directly. For realistic multi-site data we use natural partitions where they exist: the three GEO aneurysm cohorts (GSE13353, GSE15629, GSE122897) as three clients, OASIS versus 1000 Functional Connectomes MRI as two sites, UCI Smartphone HAR split by its 30 subjects, and Colon/Leukemia microarrays split across simulated sites. We also run METABRIC survival prediction, scored by concordance index.
Metrics are AUC or accuracy, concordance index for survival, F1 against known true features, selection stability (Jaccard similarity across seeds), total bytes and rounds, and filter rank correlation as a function of the differential-privacy budget.
Three checks decide the outcome.
- On the interaction-only synthetic data, the reserve variant must recover the hidden features while the hard-cut variant and the filter-only methods cannot.
- Our method must match the GA funnel's accuracy at no more than one tenth of its communication.
- The no-filter federated-gates ablation must fall behind as $p$ grows from hundreds to tens of thousands. If it does not, the funnel claim fails.
References
- LESS-VFL: Communication-Efficient Feature Selection for Vertical Federated Learning
- Vertical Federated Feature Screening
- MSEs Credit Risk Assessment Model Based on Federated Learning and Feature Selection
- Secure and Lightweight Feature Selection for Horizontal Federated Learning
- Federated Data and Feature Selection by Generalized CUR Decomposition
- Causal Multi-Label Feature Selection in Federated Setting
- Feature Selection using Stochastic Gates
- A multi-filter enhanced genetic ensemble system for gene selection and sample classification of microarray data
- A novel community detection based genetic algorithm for feature selection
- Distributed Genetic Algorithm for Feature Selection
- Screening key genes for intracranial aneurysm rupture using LASSO regression and the SVM-RFE algorithm