A model trained on clean laboratory images, such as excised leaves on a uniform background or well-lit dashcam frames, loses much of its accuracy once it runs in the field under shadows, glare, blur and clutter. The standard remedy is test-time adaptation: the deployed model keeps updating a few of its own parameters on the unlabeled images it sees. The weak point is one number, the adaptation step size. If it is too small, nothing changes. If it is too large, the model "silently collapses": it becomes confident while predicting nearly everything wrong. Field conditions drift over hours and terrain, so no single step size fits the whole stream, and nobody in a field has labels to tune it.
We propose to set the step size online with a feedback controller, a small rule that raises or lowers the step size at every batch based on a measured signal. The controller has one design choice that matters most. It is not driven by the loss the model is minimizing, because that loss keeps looking better as the model collapses. It is driven by two signals the adaptation does not optimize. The first is a label-free check of whether predictions on the incoming field images are becoming more stable under small perturbations. The second is a brake: accuracy on a few dozen labeled laboratory images stored on the device. A collapsed model fails on those images, so their accuracy drops before field accuracy is ever measurable. In short, the controller adapts as fast as the field signal rewards, and no faster than the lab reservoir tolerates.
Core Claim
Our claim is that on drifting laboratory-to-field streams, a step-size controller driven by signals separate from the adaptation loss and anchored by a lab reservoir will outperform every fixed step size chosen without target labels. We also expect it to match or beat a fixed step size tuned with about 100 labeled target images. It should do this at close to the cost of plain adaptation of the normalization layers, which keeps it feasible on the small networks used in agricultural and robotic deployment.
The most important prior result here is Realistic Evaluation of Test-Time Adaptation Algorithms: Unsupervised Hyperparameter Selection. It benchmarks label-free criteria for choosing adaptation hyperparameters, including source accuracy, prediction entropy, perturbation consistency and soft neighborhood density. It finds that algorithm rankings flip under these criteria, and that as few as 100 labeled target samples recover nearly all oracle performance. That paper only chooses static settings over pre-collected batches. We move the choice inside a single streaming pass. The paper also constrains our design. It shows that source accuracy is a poor selection objective, because it rewards not adapting at all. We therefore never let the controller maximize lab accuracy. We use it only as a one-sided brake that activates when lab accuracy falls more than a tolerance below its starting value.
On the other side, Parameter-free Online Test-time Adaptation (LAME) avoids the step-size problem by freezing all weights and refining only output probabilities. It reports up to 15% absolute gains on non-i.i.d. streams, at twice TENT's speed and half its memory. LAME itself concedes that output refinement cannot repair corrupted features. This matters for field models: A Large-Scale In-the-wild Dataset for Plant Disease Segmentation reports MobileNetV3 at 10.22% mIoU on in-the-wild PlantSeg images, against 46.24% for ConvNeXt-L. That is evidence that the features of small field models are what fails. Our proposal accepts LAME's diagnosis that uncontrolled weight updates collapse. Its answer to that diagnosis is control, not freezing.
Research Program
The adaptation itself stays deliberately simple. We use entropy minimization on the scale-and-shift parameters of the normalization layers, as in TENT, because the evaluated methods all build on it and it is cheap. The new part is how the step size $\eta_t$ is set at batch $t$. We use a proportional-integral update in log space:
$$\log \eta_{t+1} = \log \eta_t + k_p\, g_t + k_i \sum_{\tau \le t} g_\tau - \kappa\, \max(0,\; a_0 - a_t - \delta).$$
The terms are as follows. The field signal $g_t$ is the change in the model's perturbation consistency on the current field batch relative to a running average, where consistency means agreement between predictions on an image and a lightly perturbed copy. The lab signal $a_t$ is accuracy on the stored lab reservoir, re-checked every few batches, and $a_0$ is its value before deployment. The tolerance $\delta$ is the allowed lab-accuracy loss, and $k_p, k_i, \kappa$ are controller gains.
The gains and $\delta$ are fixed once, on development streams built from CIFAR-100-C and ImageNet-C corruptions chained in gradually shifting order. They are never touched on field data. The structure borrows from A PID Controller Approach for Adaptive Probability-dependent Gradient Decay in Model Calibration. That work steers a training hyperparameter using calibration error on labeled validation data at epoch boundaries. We move the loop to per-batch updates at deployment time and replace labeled validation with the two signals above.
The broader online-tuning literature supplies motivation but not a usable method, for a specific reason. Forward and Reverse Gradient-Based Hyperparameter Optimization, Scalable Gradient-Based Tuning of Continuous Regularization Hyperparameters, Self-Tuning Networks and Learning Data Augmentation with Online Bilevel Optimization for Image Classification all show that hyperparameters adapted within one run beat static sweeps. All of them differentiate through a clean labeled validation split, which does not exist in the field. Mechanic: A Learning Rate Tuner needs no validation set, but it tunes the step size to reduce the loss being optimized. For entropy-based adaptation, that loss falls fastest along the path to collapse. We treat this as a testable prediction, not an assumption.
Everything is built on edge-scale backbones: MobileNetV2/V3 for leaves, and the DRN-D38 segmenter used by WildDash for driving. We measure wall-clock time and peak memory against unmodified TENT. The reservoir check is the only added cost, and we expect it to be a small fraction of a forward pass.
Baselines
The naive reference is the frozen lab-trained model with no adaptation. For driving, the reported point is the Cityscapes-trained DRN-D38 at 16.5 mIoU on WildDash, against about 64 mIoU on Cityscapes validation (WildDash - Creating Hazard-Aware Benchmarks). For leaves, MobileNetV2 reaches 94.32% validation accuracy on lab PlantVillage images (Plant Leaf Disease Detection using MobileNetV2). We found no reported result on Lacuna for a PlantVillage-trained model evaluated on PlantSeg images, so we will measure it ourselves.
The strongest label-free competitor is TENT, EATA or MEMO with hyperparameters chosen by the best unsupervised criterion from the Realistic Evaluation study. We rerun that selection protocol on our development streams with the same adapted parameters, the same backbone and the same number of candidate settings as our controller's gain search. The strongest label-using competitor is the same algorithms tuned on 100 labeled target images, which that paper reports "recovers nearly all" oracle performance. The file gives no single numeric value for this, so we will measure it on our streams. Beating the label-free baseline on stream-averaged accuracy or mIoU is the minimum bar. Matching the 100-label baseline without any target labels is the result that would make the method worth using.
We also run LAME with its published defaults, at equal compute, and MEMO with per-step reset as a collapse-proof reference. MEMO is reported at 10–50x slower, and we will report its latency alongside accuracy.
Three ablations isolate the key ingredient. The first drives the controller with the adaptation loss itself, in the way Mechanic would; we predict it collapses. The second keeps only the field signal and removes the brake. The third keeps only the brake. We also compare against the single best fixed step size that the controller's own signals would pick offline, which separates the value of being online from the value of the signals.
Why not just…?
Why not just label 100 field images and pick a fixed step size? This is the most serious threat. The Realistic Evaluation study says it nearly closes the gap to oracle tuning, and a grower or fleet operator can often afford 100 labels. Our case rests entirely on drift. When shadows, solar angle or weather change within a stream, a step size tuned on one window should be wrong for later ones. We will test this directly by comparing both methods on streams ordered by drift and on shuffled streams. If the fixed 100-label choice wins on drifting streams too, the proposal's main claim is false. In that case the contribution reduces to a label-free approximation of it, and we will report it that way.
Why not just freeze the network and use LAME? LAME is safe and fast. But by its own account it cannot fix features, and the 10.22% versus 46.24% PlantSeg gap between small and large backbones points to features as the failure. The LAME comparison will settle this. If LAME matches our method on PlantVillage-to-PlantSeg, then feature adaptation is not buying anything for this shift.
Why not just reset the model every so often? Periodic reset is cheap and bounds collapse. It also discards useful adaptation on every reset, and its interval is itself an untuned hyperparameter. We will tune the reset interval on the same development streams and report both worst-window accuracy and average accuracy. A reset baseline that ties us on average accuracy but not on worst-window accuracy would still leave room for the controller. A tie on both would not.
Why not just train with heavier augmentation offline? Diverse training data helps but does not close the gap. Training on 18,000 varied Mapillary images still gives only 30.7 mIoU on WildDash (WildDash - Creating Hazard-Aware Benchmarks). Offline augmentation and test-time control are complementary, so we will apply our controller on top of an augmentation-heavy source model as well as a plain one.
Proposed Experiments
We run three stream families. The first is development only: chained CIFAR-100-C and ImageNet-C corruptions with gradually increasing severity, used to fix the controller's gains once. The second is the Cityscapes-to-WildDash shift, where only the 70 public validation cases have labels available to us. This makes the streams short and tests how quickly the controller responds. The third is PlantVillage-to-PlantSeg classification, restricted to plant-disease pairs present in both datasets; the size of that overlap must be checked first. These streams are built in two orders: randomly, and sorted by image brightness and blur, to simulate gradual field drift.
Metrics are accuracy or mIoU averaged over the stream, accuracy in the worst sliding window (our collapse measure), accuracy in the final window, latency and peak memory. The decisive checks are four. The controller must beat the best label-free fixed setting on the drifting streams. The loss-driven ablation must collapse where the full controller does not. On shuffled, stationary streams the controller must be no worse than the best fixed setting. Finally, the comparison against the 100-label fixed baseline on drifting streams determines whether online control is worth doing at all.
References
- Realistic Evaluation of Test-Time Adaptation Algorithms: Unsupervised Hyperparameter Selection
- Parameter-free Online Test-time Adaptation
- A Large-Scale In-the-wild Dataset for Plant Disease Segmentation
- A PID Controller Approach for Adaptive Probability-dependent Gradient Decay in Model Calibration
- Forward and Reverse Gradient-Based Hyperparameter Optimization
- Scalable Gradient-Based Tuning of Continuous Regularization Hyperparameters
- Self-Tuning Networks: Bilevel Optimization of Hyperparameters using Structured Best-Response Functions
- Learning Data Augmentation with Online Bilevel Optimization for Image Classification
- Mechanic: A Learning Rate Tuner
- WildDash - Creating Hazard-Aware Benchmarks
- Plant Leaf Disease Detection using MobileNetV2