Learning When to Look: Freshness-Aware Sensing Policies That Transfer from Simulation

Robots increasingly send their sensor readings over wireless links to an off-board computer, or wake battery-powered sensors only occasionally. Each reading then arrives with some age of information: the time since that reading was taken. Two research communities handle this badly, and in different ways. Networking research decides when to sample and transmit so that data stays fresh on a limited energy budget, but it scores freshness by the clock alone. It ignores whether the controlled system is drifting slowly or about to fall over. Robot learning research trains control policies in randomized simulators so they work on real hardware without fine-tuning, a practice called domain randomization. When those policies face delays, they only absorb stale data passively. They never decide when fresh data is worth paying for.

We propose to train two things together in simulation: a control policy and a policy that decides when the robot should request a new observation. Training randomizes both the physics and the statistics of the communication channel. During training, the critic (the learned value estimator) sees the true current simulator state. That lets us measure how much a fresh observation is worth in the current situation, which gives a learned, model-free version of the "value of information" idea from networked control. The claim is that the robot learns to spend its sensing and radio budget during the moments that are dynamically dangerous, and that this behavior transfers zero-shot to channels and physical parameters it never saw.

Core Claim

At a matched budget of observation requests, a request policy that conditions on the robot's state as well as the age of its latest observation should achieve higher task return than any age-only threshold rule. The margin should grow with how unstable the task dynamics are. Training over a family of randomized channel models should make this advantage survive when the policy is deployed on held-out real wireless traces. The contribution is three things together: making the sensing decision an action, trained in simulation against privileged fresh state, under randomization of the channel. Randomizing delays alone is not the contribution. That practice is already implicit in sim-to-real work, and random-delay RL already trains on real WiFi traces.

Where This Sits

Three strands of prior work set up the problem.

The AoI literature supplies the decision structure. Joint Status Sampling and Updating for Minimizing Age of Information in the Internet of Things formulates sampling and transmission as a constrained Markov decision process solved with a Lagrangian. It proves the optimal policy has multiple thresholds and reduces average AoI by up to 33% versus sending as fast as possible. Age-Threshold Slotted ALOHA for Optimizing Information Freshness in Mobile Networks shows that "stay silent until age crosses a threshold" halves how fast AoI grows with node density. We reuse the Lagrangian constrained formulation. We also treat age thresholds as the main competitor, not as a straw man. Both papers, however, optimize age itself, not the performance of a controlled system.

The motivating gap comes from Age-of-Information vs. Value-of-Information Scheduling for Cellular Networked Control Systems. It compares two schedulers over linear time-invariant plants:

  • an age-fair scheduler, which spreads freshness evenly across control loops;
  • a value-of-information scheduler, which prioritizes updates that most reduce the estimation error of a Kalman-style state estimator.

Estimation error plateaus with age for stable plants but grows exponentially for unstable ones. Once the network is crowded, the age-fair scheduler lets total error explode. That result argues against pure AoI objectives. It also relies on known linear dynamics and error covariances, which a contact-rich legged robot does not have. We replace the analytic value of information with a quantity read off a learned critic.

The delay-RL literature supplies the control-side method and constrains the design.

All three treat delay as something that happens to the agent. None lets the agent choose when to observe, and none randomizes physics jointly with the channel.

The sim-to-real work sets the transfer standard.

Research Program

The agent's action at each control step has two parts: a motor command $a_t$ and a binary request $u_t \in \{0,1\}$ for a fresh observation. The actor sees three inputs: the most recent delivered observation, its age $\Delta_t$ (control steps since that observation was sampled), and a short buffer of past actions. A request costs energy and succeeds or fails according to the channel. When a request succeeds, it delivers a reading with a random transport delay.

The training objective is

$$\max_\pi \; \mathbb{E}\Big[\sum_t \gamma^t \big(r_t - \lambda\, u_t\big)\Big] \quad \text{subject to} \quad \mathbb{E}[u_t] \le B,$$

Here $r_t$ is the task reward, $\gamma$ the discount, $B$ the allowed request rate per step, and $\lambda$ a Lagrange multiplier. We update $\lambda$ by dual ascent, following the constrained formulation of Joint Status Sampling and Updating for Minimizing Age of Information in the Internet of Things.

The critic receives the true current simulator state along with the actor's inputs. From it we read a learned value of an update, $Q(\cdot, u{=}1) - Q(\cdot, u{=}0)$. This gap should be large exactly when stale information is dangerous, such as the flight phase of a gallop or a hopper near a fall. We will check this by plotting the gap against simple instability proxies, such as angular velocity of the torso, to see whether the network recovers the stable-versus-unstable distinction that the value-of-information paper derived analytically.

Each episode draws both physics and channel parameters. Physics follows the mass, friction and motor-strength ranges standard in locomotion transfer. The channel draw includes:

  • the class of loss process: independent losses, bursty two-state Markov losses, or loss rates that depend on density;
  • the mean and variance of transport delay;
  • the per-request energy cost.

Held out from training are the real WiFi delay traces used by Reinforcement Learning with Random Delays and physics values outside the training ranges. The work runs in Isaac Gym, whose GPU throughput of up to 700K steps per second (Isaac Gym: High Performance GPU Based Physics Simulation For Robot Learning) makes the joint physics-channel sweep affordable. It also runs on the MuJoCo tasks used by the delay-RL papers, so our numbers are directly comparable to theirs.

Baselines

The naive reference point is zero-wait sampling: request every step, paired with a policy trained with delay randomization and history augmentation. It fixes the attainable return when budget is unlimited, and it shows how badly performance collapses once the budget constraint is imposed and requests must be dropped at random.

The first strong baseline is fixed-rate sampling. We sweep the sampling period and train a separate delay-robust controller for each period. This traces a return-versus-request-rate curve that any adaptive method must dominate.

The strongest competitor is an age-threshold trigger: request only when $\Delta_t$ exceeds a threshold, the structure proven optimal for pure freshness objectives in Joint Status Sampling and Updating for Minimizing Age of Information in the Internet of Things and Age-Threshold Slotted ALOHA for Optimizing Information Freshness in Mobile Networks. Its controller is trained with our full recipe: privileged critic, history augmentation, and joint randomization. The only difference from our method is therefore whether the trigger may look at state. We tune the threshold with the same number of training runs and seeds that we spend tuning $\lambda$ and $B$.

The controller component common to all methods is the delay-reconciled actor-critic of Addressing Signal Delay in Deep Reinforcement Learning. The file reports a 20% performance drop on the 376-dimensional Humanoid under delay for that paper's state-augmentation variant, against collapse for plain SAC and TD3. For Reinforcement Learning with Random Delays, the file reports no numeric result on Lacuna. We will rerun it on the same traces rather than quote it.

Beating the age-threshold baseline means two things. First, a higher area under the return-versus-request-rate curve across seeds, with non-overlapping confidence intervals on at least the unstable tasks. Second, holding that advantage on the held-out WiFi traces.

The ablations isolate the key ingredients:

  • removing state from the trigger input, which recovers the age-threshold family learned rather than hand-set;
  • replacing the privileged critic with a symmetric one;
  • training on a single channel model instead of the randomized family;
  • removing the age input $\Delta_t$ from the actor.

Why not just…?

Why not just calibrate the real link's delay and inject it, as the Minitaur work did? Calibration worked in Sim-to-Real: Learning Agile Locomotion For Quadruped Robots because its latency came from a fixed-rate onboard loop. Wireless delay is random. Addressing Signal Delay in Deep Reinforcement Learning finds that methods tuned to fixed delay degrade under random delay. Calibration also says nothing about when to spend energy. We include a "calibrated to training-mean delay" controller as an extra point on the fixed-rate curve. It should fail specifically on the bursty held-out traces.

Why not just use an age threshold, which theory says is optimal? It is optimal for minimizing age. It is not optimal for task return on a plant whose sensitivity to staleness changes with its state. This is the real threat to the proposal. On near-stable tasks such as Fetch reaching, we expect an age threshold to match our method, and we will report that honestly. The proposal survives only if the gap opens on Hopper, Humanoid and quadruped galloping. The per-task comparison at matched request budget settles it.

Why not just feed stale observations to a recurrent network and let it cope? A recurrent controller handles the control side. It does not decide when to request data. Prior results also favor history-augmented feedforward actors under delay (Addressing Signal Delay in Deep Reinforcement Learning; Handling Delay in Real-Time Reinforcement Learning). We will nonetheless run a recurrent version of our own method, so that the sensing-decision contribution is not confounded with the choice of controller architecture.

Why not just use analytic value-of-information scheduling? It requires linear models and error covariances (Age-of-Information vs. Value-of-Information Scheduling for Cellular Networked Control Systems). We will run it on a linearized cart-pole, where it is well-defined, as a sanity check. There, our learned value of an update should approach the analytic scheduler's performance.

Proposed Experiments

The first experiment compares return-versus-request-rate curves on MuJoCo Hopper, Walker2d, HalfCheetah and Humanoid, plus Isaac Gym ANYmal. Methods are evaluated in simulation on held-out WiFi traces and out-of-range physics. The decisive check is whether our method's advantage over the age-threshold baseline, at matched budget, correlates with task instability. The instability ordering is fixed in advance: Humanoid and Hopper above HalfCheetah, and HalfCheetah above Fetch reaching. An advantage that is uniform across tasks, or absent, would falsify the mechanism.

The second experiment is a channel-transfer matrix: train on each single channel class and on the randomized family, then test on every class and on the real traces. It tests whether randomization rather than capacity drives robustness, and whether overly wide channel ranges make the trigger over-request. That over-requesting would be the conservativeness concern raised against broad randomization.

The third experiment inspects the learned value of an update. We plot the critic gap along gait cycles to see whether requests concentrate in unstable phases.

A hardware check, conditional on access, runs a small quadruped or arm with off-board inference over WiFi, with requests metered by a radio duty cycle. The reference point is the Minitaur's reported real galloping speed of 1.18 m/s against 1.34 m/s in simulation (Sim-to-Real: Learning Agile Locomotion For Quadruped Robots). The question is whether our method keeps a comparable sim-to-real ratio while making fewer requests than fixed-rate sampling.