Field-Weighted Ensemble Nyström Heads for Lab-to-Field Transfer

Plant-disease and field-robotics classifiers are usually trained on clean laboratory photos and then deployed where lighting, clutter and blur look nothing like the lab. The gap is severe. A driving segmenter trained on clean Cityscapes falls from about 64 to 16.5 mean intersection-over-union (mIoU, the average overlap between predicted and true regions) on hazard-rich WildDash footage (WildDash). Lightweight networks reach only 10.22% mIoU on in-the-wild plant lesions (PlantSeg). Changing the training loss does not help much: invariance objectives such as IRM and CORAL fail to beat plain training in 8 of 10 real-world shifts (WILDS). What deployments do have in abundance is unlabeled field images.

We propose to put those images to work through the Nyström method, which approximates a large kernel similarity matrix from a small set of "landmark" points. Two properties of the method matter here. First, landmarks need no labels. Second, the ensemble version, which averages several Nyström approximations built from different random landmark subsets, chooses its mixture weights by how well each approximation reconstructs held-out columns of the similarity matrix, and that criterion also needs no labels (Ensemble Nystrom Method). The design follows directly. We build a small kernel classifier on top of a frozen, edge-sized image backbone. Its feature space is spanned by several landmark subsets drawn from a mix of lab and unlabeled field images. The mixture weights are chosen so the ensemble best reconstructs the field data's similarity structure. The classifier itself is then refit on lab labels in closed form. In short, unlabeled field data decides which directions of feature space the classifier may use, and lab labels decide how to use them.

Core Claim

The known failure mode of a single Nyström approximation is that its random landmarks miss a region of the data, and reconstruction error there spikes (Kumar et al.). Lab-to-field shift is exactly this failure. A kernel head whose landmarks are all lab images cannot represent field images, because those images fall outside the lab data's span.

For a feature vector $x$ and landmark set $m$, the part of $x$ that the approximation cannot represent is measured by the residual

$$ \rho(x) = k(x,x) - k_{xm} K_{mm}^{+} k_{mx}. $$

Here $k$ is the kernel, $k_{xm}$ is the row of similarities between $x$ and the landmarks, and $K_{mm}^{+}$ is the pseudo-inverse of the landmark-landmark similarity matrix. The residual is zero when $x$ lies in the span of the landmark features, and it can be computed on unlabeled field data.

We claim three things:

  1. Field-side reconstruction error, whether summarized by $\rho$ or by the Frobenius reconstruction loss of Ensemble Nyström, predicts field classification error across landmark subsets.
  2. Choosing mixture weights $\mu$ to minimize that error on unlabeled field columns, then refitting the head on lab labels, closes a measurable part of the lab-to-field gap.
  3. This beats both a single larger Nyström head and simple feature re-alignment at equal compute.

Claim 1 is the risky empirical hinge, and the program tests it first.

Research Program

Method. A frozen backbone, such as MobileNetV2 as used for PlantVillage in Plant Leaf Disease Detection using MobileNetV2, produces features. We draw $p$ landmark subsets of size $m$. Some come only from lab features, some are stratified by lab class (echoing the stratified landmark diversity of Multi‐Nyström), and some are mixed with unlabeled field features.

The mixture weights $\mu$ come from the ridge-weighting problem of Ensemble Nyström:

$$ \min_{\mu}\ \Big\| K_{\cdot V} - \sum_{r=1}^{p} \mu_r \tilde K_{r,\cdot V} \Big\|_F^2 + \lambda \|\mu\|^2 . $$

Here $V$ is a random set of unlabeled field points, $K_{\cdot V}$ holds their exact kernel columns, $\tilde K_{r,\cdot V}$ is expert $r$'s approximation of those columns, and $\lambda$ is a ridge penalty. With $\mu$ fixed, the head is kernel ridge regression (or a one-vs-rest version for classification) on the combined low-rank features, fit on lab labels. It has a closed form, so re-solving at deployment costs seconds.

What this reuses and what is new. Each piece of prior work plays a specific role:

  • Ensemble Nyström (art_c4f25cd7, art_7863a1de; Ensemble Nystrom Method) supplies the weighting machinery and the $\sqrt{r}$ reconstruction bound. We do not claim that bound transfers to classification under shift. Whether it does is what the experiments measure.
  • Common-specific low-rank decomposition (CSD) motivates the design by contrast. CSD splits the last layer into a shared part plus a low-rank domain-specific part and discards the specific part at test time. We keep several low-rank components and let field data reweight them.
  • Basis-oriented low-rank transfer (BOLT) constrains the design. It shows that confining test-time adaptation to a small low-rank subspace beats unconstrained LoRA under extreme shift. However, it needs a zoo of fine-tuned models and a single static basis, whereas our subspaces come from one lab dataset plus unlabeled field images.
  • The lab-to-field adaptation method of Triple-DARE aligns lab and field feature distributions with discrepancy and triplet losses, and gains 44.6 F1 points over non-adapted models on smartphone sensor data. That shows unlabeled field data is valuable. Our method uses such data without any backbone training, which matters on robots with tight latency budgets (TomFormer).
  • WILDS constrains the approach from the other side. Because invariance losses rarely help, we deliberately avoid a new loss and change only which subspace the classifier lives in.

Analysis. We will study how field reconstruction error and field accuracy relate across individual experts, across mixtures, and as the share of field landmarks grows. We will also report memory cost: storing $p$ experts takes $p$ times the memory of one, which is the known limitation of Ensemble Nyström.

Baselines

Do-nothing reference. A linear probe (and a kernel ridge head with lab-only landmarks), trained on lab features from the same frozen backbone. This anchors the size of the gap. On Camelyon17, WILDS reports ERM at about 93% in-distribution and about 70% out-of-distribution accuracy (WILDS). That is the out-of-distribution reference we must exceed on that dataset. Note that our frozen-backbone probe may start below full ERM fine-tuning, so we report both.

Strongest simple competitor. A single Nyström head with the same total number of landmarks as the ensemble, $p \cdot m$, drawn from the same lab-plus-field pool. This isolates whether ensembling and weighting add anything beyond simply having field landmarks. Ensemble Nyström beat single Nyström on reconstruction at fixed time on SIFT-1M, reaching 10% relative Frobenius error with about 1% of columns (Ensemble Nystrom Method). However, no reported result was found on Lacuna for either method on classification under shift.

Strongest existing adaptation method. Feature-level CORAL-style alignment of field features to lab statistics before the linear probe, since it is cheap, label-free, and uses the same unlabeled data. We also run BOLT-style diagonal tuning where a model library is available. BOLT reports 67.74% average test-time adaptation accuracy versus 56.87% for zero-shot CLIP on its benchmarks (BOLT). Those are not our datasets, so they serve only as a scale reference.

Ablations. Two ablations isolate the key ingredient: uniform weights instead of field-fitted weights, and field-fitted weights with lab-only landmarks.

Fairness and success criterion. All methods share the backbone, the unlabeled field pool, the lab labels, and a tuning budget of 20 configurations each. Hyperparameters are selected on a held-out lab split, never on field labels. Beating a baseline means a mean field-accuracy improvement whose 95% bootstrap interval over 5 seeds excludes zero, on at least two of three benchmarks.

Why not just…?

Why not just use one big Nyström head with field landmarks? This is the most serious threat. If field landmarks alone do the work, the ensemble is unnecessary overhead. The equal-landmark single-head baseline tests this directly. If it matches our method, we will report that the useful ingredient is label-free field landmarks rather than ensembling, and drop the ensemble claim.

Why not just align field features to lab statistics (CORAL) and keep a linear probe? Alignment is cheaper still. However, WILDS found CORAL gave no meaningful gains on 8 of 10 real shifts (WILDS). Matching second moments also cannot recover field directions that lab data never spans, and that missing span is exactly what $\rho$ measures. Our baseline runs alignment on identical data. We also report results separately for field images with high versus low $\rho$ to see where each method wins.

Why not just fine-tune the backbone with field pseudo-labels or heavy augmentation? Augmented lab training still collapses in the field (MobileNet-class models reach 10.22% mIoU on PlantSeg), and pseudo-label fine-tuning needs on-device backprop. The two approaches are complementary rather than exclusive. We include an augmented-lab backbone as an extra row to test whether our head still adds gains on top of it.

Proposed Experiments

Benchmarks. We use three settings:

  • PlantVillage to PlantDoc. We restrict to the disease classes shared by the lab PlantVillage images and the field PlantDoc images, using leaf crops for classification. No reported cross-domain number was found on Lacuna for this transfer, so we establish the probe baseline ourselves.
  • Maize lab/field data. We re-split the combined maize dataset of the maize attention study so that training is lab-only and testing is field-only. That paper reports 98.7% on a mixed split, which is not comparable to a lab-to-field split.
  • Camelyon17 from WILDS. This is a standard, non-agricultural hospital shift with the reported ERM reference.

In every setting, a disjoint slice of field images serves as the unlabeled pool. Field labels are used only for evaluation.

Decisive check (run first). Across 100 random landmark subsets, we compute the Spearman correlation between field reconstruction error and field accuracy. If the correlation is weak (below 0.3 in magnitude), the weighting rationale fails. In that case the project reduces to the field-landmark question and is reported as a negative result.

Main comparisons. We report field accuracy and macro-F1 against all baselines. We sweep the number of experts $p \in \{1, 2, 4, 8, 16\}$ at fixed total landmarks to test whether gains follow the roughly $\sqrt{p}$ trend that Ensemble Nyström's reconstruction bound would suggest. We also sweep the share of field landmarks from 0 to 1.

Costs. We measure memory and latency on an edge-class CPU against the single head, reporting whether the $p$-fold storage cost is justified by accuracy.