Suppose many hospitals, phones, or cameras each hold private, labeled face images, and together they train one facial landmark detector. A landmark detector is a model that places points on the eyes, mouth, nose and jaw. In federated learning, only a few devices can send updates to the server in each round. The usual practice is to pick those devices at random. We propose a different rule. Every device runs a small, frozen "proxy" detector on its own data and reports one short vector: how badly that detector misses each group of facial points (eyes, brows, mouth, nose, jawline). Each group's error is weighted by how much errors there damage downstream tasks such as expression recognition. In each round the server picks the set of devices whose failure vectors together cover the widest range of failure types.
The bet is that, for structured outputs like landmarks, the useful notion of "diverse clients" is diversity in how a model breaks, not diversity in gradients or label counts. Covering failure modes during training should matter most for the goal that counts in deployment: working on camera conditions, poses and occlusions that no training device ever saw. Today nobody measures this. Federated domain generalization benchmarks contain only classification tasks, and diversity-aware client selection has only been tested under label skew.
Core Claim
Consider a proxy detector that is roughly ten times cheaper than the target model and trained once on public data. Its per-landmark-group error profile, weighted by downstream sensitivity, is a better client descriptor for federated selection than gradient similarity or pose and appearance statistics. It should lower error and catastrophic-failure rate on held-out landmark domains at a fixed per-round communication budget. The descriptor costs one forward pass per local image, and each client sends fewer than ten numbers.
This claim can fail in two separable places. First, the proxy's failure profile may not preserve the target model's ranking of which clients are similar. Second, even if the ranking holds, a cheaper descriptor such as a head-pose histogram may give the same benefit. The experiments test both.
Why this intersection, and what already exists
Diversity-aware client selection offers several mechanisms. Diverse Client Selection for Federated Learning via Submodular Maximization picks clients whose gradients best "cover" the population. Federated Generalization via Information-Theoretic Distribution Diversification picks clients at the extremes of gradient space. FedCor models correlations between client losses with a Gaussian process. An EMD-Based Adaptive Client Selection Algorithm ranks clients by distribution distance. All four describe clients through classification gradients or label histograms, and all are evaluated on label-skewed classification. We reuse the submodular selection machinery from the first paper but replace its descriptor.
Proxy-based data selection supplies the second ingredient. Selection via Proxy shows that shrunken or briefly trained models rank classification examples with Spearman correlation above 0.9 to large models. Superfiltering and S2L show the same weak-to-strong transfer for language models. Coreset Selection for Object Detection is the constraining result. Classification coreset scores (Herding, k-Center) failed to beat random selection on detection until the scoring was rebuilt around object-level structure. We take that lesson seriously: a landmark proxy has to score structure, not scalar loss. We also move the proxy's job from ranking examples to ranking clients. No work in the file tests proxy rank consistency for coordinate regression.
The choice of what to score comes from the landmark-sensitivity literature, which motivates the proposal. Impact of Facial Landmark Localization on Facial Expression Recognition finds four things:
- mean landmark error correlates poorly with downstream expression recognition;
- head pose hurts detectors more than facial deformation does;
- eye and mouth points matter far more than jawline points;
- alignment still improves a landmark-free network (ExpNet) by 23.7%.
Failure Detection for Facial Landmark Detectors shows that landmark groups differ in how informative they are about failure. It uses failure predictions only at test time, as a fallback switch. Person X Detector shows that small eye-alignment errors in training data can collapse face recognition, from 13.03% to 0.13% average precision across two identities. Together these papers say two things: an unweighted error average is the wrong summary of a client, and catastrophic localized failures are what must be covered.
Federated domain generalization supplies the evaluation regime and a warning. Federated Domain Generalization with Generalization Adjustment re-weights all participating clients by their generalization gap, but does not select a subset. Benchmarking Algorithms for Federated Domain Generalization reports three findings that shape our design:
- plain federated averaging (FedAvg) is very hard to beat;
- specialized methods collapse as client count grows toward 200, which the authors call the "scale trap";
- on CelebA, FedAvg drops from 61% to 45% as clients become single-domain.
No paper in the file joins all four pieces. The nearest neighbors are gradient-diversity selection on classification and object-level proxy coresets in centralized detection.
Research Program
We first build a federated landmark generalization benchmark, because none exists. WFLW training images are grouped into domains using their pose, expression, illumination, make-up, occlusion and blur labels, supplemented with 300W. The images are distributed over 10, 50 and 200 clients. Domain mixing is controlled by the heterogeneity parameter λ from the FedDG benchmark paper, so that λ = 0 means every client holds a single domain. Each run holds out one entire domain from all clients, such as large pose or heavy occlusion, and tests on it. COFW serves as an additional unseen test set.
Next we build the downstream sensitivity weights, once, on the server with public data. We perturb each landmark group of ground-truth annotations by controlled amounts, then measure how much a fixed downstream model degrades. The downstream models are an action-unit detector that crops regions around landmarks, in the style of Action Unit Detection with Region Adaptation, evaluated on BP4D, and an expression classifier evaluated on CK+. The resulting per-group weights $w_g$ should reproduce the eye/mouth-over-jaw ordering reported by the expression-recognition study. If they do not, that is itself informative.
Each client $k$ then runs the frozen proxy $f_0$, a MobileNet-scale heatmap detector trained on 300W, over its local labeled data. It sends one number per landmark group $g$, from $g = 1$ up to $G \approx 6$ groups:
$$p_{k,g} = w_g \cdot \frac{1}{|D_k|}\sum_{x \in D_k} \min\!\big(e_g(f_0, x),\, \tau\big),$$
Here $D_k$ is client $k$'s data and $e_g$ is the normalized error of group $g$ on image $x$. The clip $\tau$ caps each error so that one grotesque failure counts as "a failure" rather than dominating the average. A second variant sends the fraction of images whose group error exceeds $\tau$. A label-free variant replaces $e_g$ with the spread of the proxy's heatmaps.
Each round, the server chooses $K$ clients by greedily maximizing a facility-location coverage objective over the profiles $p_k$ of all available clients. Profiles are refreshed every $R$ rounds, which keeps the communication cost negligible. The target detector is trained with ordinary FedAvg on the selected clients, so selection is the only thing that changes.
Baselines
The naive reference is FedAvg with uniform random selection of $K$ clients per round. The FedDG benchmark paper shows this is a demanding bar, especially at 200 clients. It is run with the same $K$, the same local epochs and the same round budget as our method. Early stopping is chosen on a validation split of the source domains, never on the held-out domain. This respects that paper's finding that out-of-domain accuracy often peaks early. Beating it means lower held-out-domain error and lower failure rate (the share of faces above 10% error) across three seeds at the same budget.
The strongest existing selection method is gradient-coverage submodular selection from Diverse Client Selection for Federated Learning via Submodular Maximization, applied directly to the landmark regression loss. FedCor and EMD-based selection are secondary comparisons. The gradient method is the most direct competitor because it uses the same greedy coverage machinery, so any difference isolates the descriptor. It is given the same $K$ and its native gradient-refresh schedule. We also report its communication cost, since it requires gradient snapshots from candidate clients. Beating it means matching or improving held-out error while sending orders of magnitude fewer numbers.
The strongest aggregation competitor is Generalization Adjustment added to random selection. On DomainNet it raised FedAvg from 57.29% to 60.24% average accuracy (Federated Domain Generalization with Generalization Adjustment). We run it with its variance regularizer tuned over the same number of trials as our $\tau$ and $R$. Because it acts on aggregation weights rather than selection, we also test it combined with our method. We found no reported result on Lacuna for any federated landmark benchmark, and no reported federated number for these methods on WFLW. As a ceiling, we train the target architecture centrally on the pooled source domains. We keep centralized full-WFLW results in view for scale: 4.33% error for Gaussian Vector and 4.62% for DeCaFA.
Three ablations isolate the key ingredient:
- Pose-and-appearance coverage. Clients send a histogram of proxy-estimated yaw, pitch and brightness instead of error profiles; selection is otherwise identical.
- Unweighted profiles. The error profile is kept but $w_g = 1$ for all groups, isolating the downstream-sensitivity weighting.
- Oracle profiles. Profiles are computed by the current target model rather than the frozen proxy, bounding what a better proxy could buy.
Why not just…?
Why not just use random FedAvg and train longer? The benchmarking paper found that FedAvg often beats elaborate methods, and it is a genuine threat. Its weakness is structural. Under single-domain clients, random rounds are dominated by the most common domains, and CelebA accuracy fell from 61% to 45% as heterogeneity rose. Training longer does not fix this and can overfit source domains. The λ sweep settles the question. We expect no gain at λ = 1 and a gain growing toward λ = 0. If the gap stays within seed noise at λ = 0, the proposal fails.
Why not just select clients by pose and lighting statistics? This is the most serious alternative. Pose is the dominant cause of landmark failure, so a yaw histogram may capture most of what a failure profile captures. We would not argue around this result. The pose-coverage ablation reports it directly. The failure profile should win specifically on held-out occlusion, blur and make-up domains, where failures are not explained by pose. If it wins only on the large-pose domain, the honest conclusion is that metadata coverage suffices, and we would report that.
Why not just use gradient-diversity selection, which already exists? Gradients of an averaged landmark loss are dominated by the many jawline points and by easy images. The sensitivity literature says these matter least downstream. Gradients also change round to round and cost full model-sized uploads. The direct comparison measures error, failure rate, downstream F1 and bytes sent. The result is a draw at far lower cost, or a win.
Why not just select by plain landmark error? The expression-recognition study shows that mean error is a weak predictor of downstream success. The unweighted-profile ablation tests whether that finding carries into selection. We evaluate downstream action-unit F1 on BP4D and expression accuracy on CK+ using the federated detector's landmarks, not only landmark error.
Decisive Checks
The first check is proxy validity, the regression analogue of the rank-correlation test in Selection via Proxy. We compute the Spearman correlation between client-to-client similarities from proxy profiles and those from oracle target-model profiles, at several points during training. If it falls well below the >0.9 seen in classification, then the proxy is the bottleneck. The oracle ablation then tells us how much is lost.
The second check is held-out-domain landmark error and failure rate at equal rounds and equal $K$, across λ ∈ {0, 0.5, 1} and 10 to 200 clients. The scale-trap curve is the key figure. A selection rule that prunes redundant single-domain clients should flatten the collapse that specialized methods show as clients grow. A rule that chases extremes could steepen it. Either outcome answers an open question from the FedDG benchmark.
The third check is downstream utility: BP4D action-unit F1 and CK+ expression accuracy, computed through landmark-based crops. The claim that sensitivity weighting matters stands only if the weighted profile beats the unweighted one here, even when their landmark errors are similar.
References
- Diverse Client Selection for Federated Learning via Submodular Maximization
- Federated Generalization via Information-Theoretic Distribution Diversification
- FedCor: Correlation-Based Active Client Selection Strategy for Heterogeneous Federated Learning
- An EMD-Based Adaptive Client Selection Algorithm for Federated Learning in Heterogeneous Data Scenarios
- Selection via Proxy: Efficient Data Selection for Deep Learning
- Superfiltering: Weak-to-Strong Data Filtering for Fast Instruction-Tuning
- SmallToLarge (S2L): Scalable Data Selection for Fine-tuning Large Language Models by Summarizing Training Trajectories of Small Models
- Coreset Selection for Object Detection
- Impact of Facial Landmark Localization on Facial Expression Recognition
- Failure Detection for Facial Landmark Detectors
- Person X Detector
- Federated Domain Generalization with Generalization Adjustment
- Benchmarking Algorithms for Federated Domain Generalization
- Action Unit Detection with Region Adaptation, Multi-labeling Learning and Optimal Temporal Fusing
- Gaussian Vector: An Efficient Solution for Facial Landmark Detection
- DeCaFA: Deep Convolutional Cascade for Face Alignment In The Wild