Farmers and regulators who manage fertilizer, water or fish stocks cannot see the system they control. They pay for glimpses of it: a satellite image that may be cloud-covered, a soil probe that slowly drifts out of calibration, or a destructive biomass sample that is accurate but expensive. Recent work in reinforcement learning (RL), where an agent learns a decision policy by trial and error in a simulator, has shown that an agent can learn both how much nitrogen to apply to wheat and when it is worth paying to measure the crop. That agent was trained and tested in a simulator that returns perfect readings whenever it measures. Real readings are not perfect, and the agent's measuring habits rest directly on that assumption: it learned to buy cheap proxies precisely because, in simulation, cheap proxies never lie.
We propose to train the measuring-and-fertilizing agent across many randomized versions of the simulator. Each version differs in crop and soil physics and also in how each sensor fails, through cloud dropouts, noise, drift and delivery delay. The aim is a policy that works without retraining on data streams whose failure modes it has never seen exactly. The robotics technique being borrowed is called domain randomization: train across a wide family of simulated worlds so the real world looks like one more member of the family. Our claim is that in environmental monitoring, randomizing how observations fail matters more than randomizing the physics. It changes which sensors the agent buys and how often it re-measures, and a policy trained on clean observations cannot recover that behaviour by adding a denoising filter at deployment.
Core Claim
To Measure or Not built a cost-sensitive crop environment on the WOFOST crop growth simulator with 33 years of NASA POWER weather. Its LSTM-PPO agent chooses nitrogen doses and, at each step, which of six crop and soil features to purchase. It reaches a median yield of 7.46 t/ha with about five strategic measurements a year, against 7.30 t/ha for a fixed standard-practice schedule. The authors explicitly name the gap between simulation and reality (noise, weather interference, calibration drift) as unaddressed, and their crop parameters are fixed.
We claim three things. First, a measurement policy trained on clean observations is badly matched to unreliable sensors. It will over-trust a single noisy cheap reading and under-purchase confirming measurements. Second, training across randomized, physically grounded sensor-failure models produces a policy that infers a channel's current reliability from its own history and shifts purchases toward more reliable channels when needed. Third, this shift in when and what to measure is the part a post-hoc filter cannot supply. The filter fixes how readings are interpreted but leaves the clean policy's purchasing schedule unchanged.
The idea differs from the closest work in direct ways. To Measure or Not supplies the environment and the baseline agent, but it has no noise and no randomization. The robotics randomization papers, Sim-to-Real quadruped locomotion and (CAD)$^2$RL, supply the method, but their agents observe for free at a fixed rate. Here observation is itself a priced decision that randomization reshapes.
Research Program
We extend the WOFOST winter wheat environment in two ways. The first replaces the single feature mask with source-specific measurement channels, so the problem becomes a small instance of multi-source data fusion. These channels are a cheap satellite-style leaf area index that has cloud dropouts and a multi-day delivery delay, a cheaper but drifting in-situ soil moisture probe, an accurate but costly ground leaf area index survey, and an expensive near-exact biomass sample. The motivation is the column-versus-ground and occlusion problems documented in multi-source environmental work. The Barents/Kara chl-a study shows how persistent cloud blinds satellite channels, and the ecosystem informatics review models sensor faults as hidden working/failed states.
The second extension is a randomization distribution with two parts. The dynamics part perturbs WOFOST soil hydraulic and nitrogen parameters and cultivar parameters within agronomically plausible ranges, and samples weather years. The observation part draws, per episode, each channel's noise scale, dropout rate, drift rate, delay and fault probability. Cloud-dropout statistics should be fitted to real revisit-gap records rather than chosen by hand.
Two failure modes reported for randomization in robotics constrain the design. One is the "freezing" problem, where an overly wide distribution yields ultra-conservative policies; here that would mean an agent that stops measuring or over-fertilizes defensively. The other is unphysical parameter samples. We therefore restrict ranges to physically valid bounds and compare narrow, empirically fitted and wide ranges, rather than assuming wider is better.
The agent is a recurrent actor that sees only purchased, corrupted readings and their timestamps. It is trained with an asymmetric critic, a value network that during training also sees simulator-only information. Following Unbiased Asymmetric Actor-Critic, the critic conditions on both the observation history and the privileged state. That paper proves a critic conditioned on state alone biases the policy gradient, and shows it collapsing in practice. Following Informed Asymmetric Actor-Critic, we test whether feeding the critic everything, meaning the full crop state plus every channel's latent reliability parameters, hurts relative to a selected subset. That paper reports full-state critics can be degraded by irrelevant features, and its own criteria guide the selection. We use the asymmetric critic because it helps learning; the central claim does not depend on it, and an ablation removes it.
"Reality" is held out in layers, because a field trial is not affordable in the first phase. One layer is weather years never used in training. Another is soil and cultivar parameter combinations outside the training ranges. A third is sensor-failure processes structurally different from those randomized, such as correlated multi-week cloud seasons, step-change probe faults and biased rather than noisy satellite leaf area index. This is sim-to-sim evidence and we will call it that. If the results hold, a later phase would replay logged real sensor streams from instrumented plots through the policy as an offline check before any live use.
Baselines
The naive reference points come from the original environment. The none-observed policy reached 7.21 t/ha median yield and the fixed standard-practice schedule reached 7.30 t/ha, as reported in To Measure or Not. Both will be re-run on our corrupted held-out test worlds. The no-cost agent's 7.86 t/ha in the same paper serves as a clean-world ceiling.
The strongest existing method is that paper's LSTM-PPO "realistic cost" agent. It reports 7.46 t/ha median yield at about five measurements per year, but only on noise-free observations. No reported result was found on Lacuna for that agent under sensor corruption or held-out soil parameters, so we will produce that number ourselves. We re-implement it in our multi-channel environment with clean observations, using the same network size, environment-step budget and hyperparameter search budget as our agent. We then evaluate it on the identical held-out test worlds.
The second strong competitor is that clean-trained agent combined at deployment with a particle filter. The filter would use WOFOST as its process model and would be given the true test-time noise model, a deliberately generous advantage. A system-identification baseline calibrates WOFOST to each held-out site from a season of synthetic observations and then trains a policy on that single calibrated simulator.
Ablations isolate the key ingredient. One randomizes dynamics only and another randomizes observations only. A third replaces structured failures with independent Gaussian noise of matched variance. A fourth compares narrow, fitted and wide ranges. A fifth compares the history-plus-state critic, the full-privilege critic, the selected-privilege critic and a symmetric critic.
Beating the strongest baseline means two things on the held-out corrupted worlds. The agent must achieve higher median cost-adjusted return, meaning yield value minus fertilizer and measurement cost, without applying more nitrogen. It must also improve worst-decile yield, with the gap larger than seed-to-seed variation across at least five training seeds. On clean test worlds it should lose no more than a small margin relative to the 7.46 t/ha reference.
Why not just…?
Why not just keep a fixed fertilization and measurement calendar? It is cheap and regulators understand it, and its weakness is already documented. Standard practice reached 7.30 t/ha against 7.46 t/ha for the adaptive agent in To Measure or Not. In fisheries, Risk-Sensitive Policies for Sustainable Renewable Resource Allocation reports constant proportional harvesting at $90.7M average and $65M worst case, versus about $113.5M and $90M for adaptive policies. However, a calendar does not depend on sensors at all, so it may degrade least under corruption. We will report it on every test world. If the clean-trained adaptive agent falls below the calendar under corruption, that alone supports the motivation.
Why not just add Gaussian noise to observations during training? This is the most likely way for the proposal to lose. If matched-variance independent noise recovers most of the gain, then structured, source-specific failure modelling is unnecessary and we will say so. Our prediction is that independent noise teaches the agent to average repeated readings. It does not teach the agent to detect a drifting probe or a cloud season and then switch channels. We will check this directly in the decisive test below on failure processes absent from training.
Why not just train on clean observations and add a filter at deployment? A filter repairs interpretation. The clean policy still decides what and when to buy as if every reading were exact, so it will keep buying a channel the filter has learned to distrust. We give the filter the true noise model, which our agent never sees, so this comparison is biased against us. If the filter baseline still matches us, the contribution reduces to filtering and we would report that.
Why not just calibrate the simulator to each site and optimize there? Calibration needs observations from the very sensors whose failures are the problem, and sustainability systems are described as radically uncertain and sparsely observed (Computational Sustainability). Robotics evidence suggests diversity can beat fidelity ((CAD)$^2$RL), but also that fidelity plus randomization worked best for legged robots (Sim-to-Real quadruped). We therefore also test calibration followed by narrow randomization around the calibrated point, and treat that hybrid as a legitimate possible winner.
Decisive Checks
The first decisive check is the behavioural one. On held-out worlds where the satellite channel suffers a multi-week cloud season or the soil probe develops an unannounced step fault, does the randomized agent shift spending to the ground survey or biomass channel within a few steps, while the clean agent keeps buying the failed channel? We measure this as the fraction of measurement budget spent on the degraded channel after fault onset. We expect the result to explain the yield gap: return differences should concentrate in exactly those episodes.
The second check sweeps randomization width. If returns fall and measurement counts drop toward zero as ranges widen, we have reproduced the freezing problem in an agronomic setting. That would argue for fitted ranges and would bound how far this approach can be pushed.
The third check is the critic ablation, which tests whether IAAC's finding that full privileged information hurts carries over when the privileged signals are physical crop states and sensor-reliability parameters.
Results will be reported as median, worst-decile, nitrogen applied and measurements per season. Each is broken out by held-out layer (weather, soil, failure type) so that a gain in one layer cannot mask a loss in another.
References
- To Measure or Not: A Cost-Sensitive, Selective Measuring Environment for Agricultural Management Decisions with Reinforcement Learning
- Sim-to-Real: Learning Agile Locomotion For Quadruped Robots
- CAD2RL: Real Single-Image Flight Without a Single Real Image
- Machine learning-based modeling of chl-a concentration in Northern marine regions using oceanic and atmospheric data
- Machine Learning in Ecosystem Informatics and Sustainability
- Unbiased Asymmetric Actor-Critic for Partially Observable Reinforcement Learning
- Informed Asymmetric Actor-Critic: Leveraging Privileged Signals Beyond Full-State Access
- Risk-Sensitive Policies for Sustainable Renewable Resource Allocation
- Computational Sustainability