Federated Coarse-Label Mapping on Frozen Features with Client-Local Label Models

Detailed land-cover maps at one-meter resolution are expensive to produce. Hand-mapping the Chesapeake Bay watershed cost $1.3 million and took ten months of labor. A cheaper route already exists: train the fine-resolution model only against free, coarse 30-meter land-cover labels. The model's per-pixel predictions are averaged over each coarse cell, and the model is penalized when that average disagrees with the coarse label. In practice, though, the high-resolution imagery that would make this useful worldwide is often licensed per region, owned by municipalities, or collected by NGOs and farms that will not pool it. We propose training such a mapper by federated learning, in which institutions keep their imagery and exchange only model updates. The heavy image encoder is a frozen, pre-trained "foundation" network, so the only thing trained and transmitted is a small prediction head. The coarse labels are public, the imagery stays private, and the shared model is small.

The obvious version of this idea (FedAvg on top of frozen features, using the coarse-cell averaging loss) runs into two problems. Our proposal is built around them. First, frozen vision encoders produce one feature per 14–16 pixel patch, which blurs exactly the road and building edges that a downsampling-free branch was needed to recover in centralized work. Second, the coarse labels are wrong in region-specific ways. They are out of date where land has changed, and the mapping from a coarse class like "developed, low intensity" to fine classes differs between Maryland suburbs and Iowa farm towns. When those regional errors are averaged across clients, they become client drift. Our central move is to give each client a small, private label model: a matrix describing how its own coarse labels relate to fine land cover. That matrix absorbs regional label bias locally, while the shared head learns what land cover actually looks like.

Core Claim

With a frozen encoder, coarse-to-fine weak supervision becomes a problem of learning from label proportions over coarse cells. In the federated version of that problem, most of the harm from non-identical clients comes from regionally biased coarse labels rather than from regionally different imagery. Concretely, for client $k$ with coarse cells $c$, coarse label $l_c$, and fine pixels $i \in c$, we train a shared head $f_\theta$ on frozen features $z_i$ together with a client-local matrix $T_k$, whose row $T_k(l)$ is the expected fine-class distribution inside a cell labeled $l$ in that region:

$$ \mathcal{L}_k(\theta, T_k) = \sum_{c} \mathrm{KL}\!\left( T_k(l_c) \,\Big\|\, \tfrac{1}{|c|}\textstyle\sum_{i\in c} f_\theta(z_i, x_i) \right) + \lambda \sum_l \mathrm{KL}\!\left(T_k(l)\,\|\,T_0(l)\right). $$

Here $x_i$ is the raw pixel, and $T_0$ is a global prior estimated once from a small labeled region, in the way the per-class statistics in Large Scale High-Resolution Land Cover Mapping With Multi-Resolution Data were estimated. The weight $\lambda$ keeps each $T_k$ from drifting into a degenerate explanation that the head could exploit. The first term is a symmetrized-in-spirit variant of that paper's "super-resolution loss": it compares averaged fine predictions against the expected fine distribution for each coarse class. Only $\theta$ is averaged on the server. $T_k$ never leaves the client and is not used at test time. The prediction is therefore shared, and only the explanation of each region's labels is personal.

The second ingredient addresses spatial fidelity. $f_\theta$ consists of a linear or two-layer classifier on the upsampled patch features and a small full-resolution refinement branch on raw pixels, a few convolution layers with no downsampling. The branch sharpens class boundaries the way a guided filter would. Both parts are federated, and the whole head is meant to be several orders of magnitude smaller than an end-to-end dual-branch network. We will measure its exact parameter count and bytes per round rather than assume them.

Relation to Prior Work

Two centralized methods supply the supervision signal and motivate the spatial-fidelity concern. Large Scale High-Resolution Land Cover Mapping With Multi-Resolution Data trained a 1 m U-Net from scratch against 30 m NLCD by matching aggregated predictions to per-class distributions. It also documented failures where classes look spectrally alike, such as desert roads. Paraformer, in Learning without Exact Guidance, trains a parallel downsampling-free CNN and Transformer end to end. Its global pseudo-label masking (PLAT) drops pixels where predictions and coarse labels disagree. Both methods assume pooled data and trainable backbones. We keep their loss family but freeze the encoder, and we replace a single global disagreement mask with per-client label models. A global mask is exactly the design the research file flags as collapsing on clients with heavily stale labels.

The federated side sets constraints. Learning Across Decentralized Multi-Modal Remote Sensing Archives with Federated Learning shows that plain FedAvg trails feature-aligned training on BigEarthNet-MM by 9.5–10.4 F1 points. Its remedy, whitening and contrastive alignment of trainable backbones under full supervision, cannot be applied to a frozen encoder. Our bet is that once the encoder is frozen and shared, the feature-side mismatch shrinks and the label side dominates. FedICRA is the closest federated weak-supervision method. It reconciles clients whose weak labels come in different formats (points, scribbles, boxes) at the same resolution, using CRF and tree-energy priors on local encoders. It neither bridges a 30× scale gap nor models how a label source is systematically biased per site. pFedES personalizes federated learning by sharing a feature extractor and keeping task heads local. We do the reverse: the head is shared and only the label model is local.

The frozen-feature literature supplies the method and the cost argument. Efficient Pre-Trained Features and Recurrent Pseudo-Labeling reports 18–21× faster training than end-to-end adaptation methods on Office-31 with frozen features, and shows that the choice of backbone matters. TwinTURBO shows that a frozen RADIOv2.5-B plus a light projector beats supervised training in low-label regimes. Both work on classification. Neither tests whether frozen patch features survive dense, sub-patch prediction under coarse supervision, and that is the risk we test directly.

Research Program

We will build the federated system on the Chesapeake Bay NAIP/NLCD benchmark, using administrative or regional partitions as clients so that the non-identical splits are geographic rather than random. Fine-resolution ground truth is used only for evaluation. Frozen backbones include RADIOv2.5-B and at least one convolutional encoder, chosen by a feature-distance criterion in the spirit of the PRPL paper. The RGB bands go to the encoder, and the near-infrared band goes only to the refinement branch. The analysis has three parts. The first measures how much accuracy freezing costs at boundaries, by comparing boundary-band IoU with and without the refinement branch. The second decomposes the federated gap into feature-side and label-side drift. To do this, we hold imagery fixed and swap in coarse labels drawn from other regions' $T$ statistics, and separately hold labels fixed and swap imagery. The third tests whether the learned $T_k$ recover real regional structure. We check this by comparing learned $T_k$ with the true fine-within-coarse distributions computed from held-out ground truth, which gives an interpretable diagnostic that end-to-end methods lack.

Baselines

The naive reference is a frozen encoder with a linear head trained by FedAvg on nearest-neighbor-upsampled coarse labels with per-pixel cross-entropy. Upsampled-label training is documented as producing blobby maps and, at a 120× gap, incoherent ones in Learning without Exact Guidance. Beating it is necessary but uninformative.

The strongest existing method is centralized Paraformer, which reports 64.65% mIoU on Chesapeake and 49.39% mIoU on the 120× Poland setting (Learning without Exact Guidance). We will rerun it on our exact tiles and splits rather than rely only on the quoted number. It has no privacy constraint and a trainable backbone, so it is an upper reference: a federated frozen method coming within a few mIoU points while transmitting a small fraction of the bytes counts as success. We also report the centralized U-Net with super-resolution loss, using its reported 93.35% accuracy and 71.32% Jaccard on held-out Iowa (Large Scale High-Resolution Land Cover Mapping) as the out-of-region reference. We will reproduce it under our splits.

The federated competitors all share the same frozen features, head architecture, number of rounds, local epochs, and hyperparameter search budget. One is FedAvg with a single global $T_0$, which is the federated super-resolution loss without our ingredient. Another is FedAvg with a global PLAT-style disagreement mask. A third is a FedRep-style variant that personalizes the head's last layer instead of the label model. A fourth is a FedICRA-style setup with site-contrastive representation, adapted to the head. Finally, isolated local training (each client alone) shows whether federation helps at all. Our method beats the field if it improves held-out-client mIoU over global-$T_0$ FedAvg and over last-layer personalization. The improvement must be statistically separated across at least three seeds and partition draws, and it must grow as label staleness is injected.

Why not just…?

Why not just centralize, since NAIP and NLCD are public? For this benchmark one could, which is why centralized Paraformer is our upper reference rather than a competitor. Chesapeake stands in for the target setting of licensed sub-meter or municipal imagery that cannot be pooled. The scientific question is how much federation plus freezing costs relative to pooling, and the centralized run answers it directly.

Why not just let each client train its own head? This is a real threat. Each region has many coarse cells, and a local head avoids drift entirely. The decisive test is held-out-client and held-out-region (Iowa) accuracy, plus performance on small clients with few tiles. If isolated training matches federation on those, the federated contribution is void and we will report that.

Why not just solve a one-shot linear proportion regression from shared statistics? With a frozen encoder, regressing coarse-class fractions linearly on cell-mean features needs only each client's feature Gram matrix and cross-moments. That gives an exact one-round federated solution with no drift. It cannot sharpen boundaries below the patch scale, and it ignores the non-linearity of averaging softmax outputs. We include it as a baseline: if it comes within a point of our method on interior-pixel mIoU, the gains we claim must come from the refinement branch and $T_k$ at boundaries and stale cells, and we will show results stratified that way.

Why not just federate LoRA fine-tuning of the backbone? It may recover boundary detail that freezing loses. We run it under an equalized per-client compute and communication budget. If it wins by a large margin at equal bytes, then freezing is the wrong trade, and the frozen-plus-refinement design is refuted for this task.

Proposed Experiments

The main experiment trains all methods on Chesapeake with regional clients and reports mIoU, boundary-band IoU (pixels within a few meters of a class edge), held-out-client mIoU, Iowa transfer, and bytes communicated. The stress test uses the Poland 120× setting, where frozen 16-pixel patches span only 4 m but labels span 30 m, to see whether the refinement branch keeps maps coherent where non-Paraformer baselines reportedly fail.

The decisive check for the label-model claim is a controlled staleness sweep. In a subset of clients, we replace a growing fraction of coarse cells with the labels of a neighboring land-use transition, simulating development since the coarse map was made. If client-local $T_k$ is the right mechanism, global-$T_0$ FedAvg and global masking should degrade steeply on those clients and on the shared model, while our method should degrade gently. The learned $T_k$ should also move measurably toward the corrupted statistics. If the curves coincide, the drift lies elsewhere, and we fall back to the feature/label decomposition analysis to locate it.

A second decisive check runs the refinement-branch ablation alongside the one-shot linear baseline. Together they tell us whether frozen features with a tiny full-resolution branch close the boundary gap that motivated Paraformer's heavy architecture, or whether that gap requires a trainable encoder.