In classical control and reinforcement learning (RL), environmental noise is frequently treated as a uniform background hum—a constant Gaussian variance applied across the entire state space. This assumption of homoscedasticity makes the mathematics elegant, but it fails fundamentally when applied to the real world. A drone navigating through a canyon does not experience uniform wind; it encounters severe turbulence near cliff edges and calm air in the open. A robot's camera does not maintain constant reliability; its accuracy plummets during sudden occlusions or lighting changes.
When a system assumes uniform noise, it suffers from severe model bias. It over-generalizes noise levels from stable regions to volatile ones, leading to either reckless overconfidence in dangerous zones or excessive timidity in safe ones. To build truly robust agents, we must model noise as a dynamic function of the system's current state—a property known as heteroscedasticity.
The Core Mechanism: Decoupling Mean Dynamics from Variance
How do we mathematically capture noise that changes depending on an agent's location or action? The core intuition is to decouple the prediction of the next state (the mean dynamics) from the prediction of the uncertainty around that state (the variance).
Instead of fitting a single function that outputs a deterministic next state plus a global variance $\sigma^2$, heteroscedastic models employ dual processes or multi-component architectures. As John D. Martin et al. demonstrate in Extending Model-based Policy Gradients for Robots in Heteroscedastic Environments, using Chained Gaussian Processes allows an agent to learn two distinct latent processes simultaneously: one that maps the state-action pair to the expected next state, and another that maps the exact same pair to the expected variance. By doing so, the agent explicitly constructs a "risk map" of its environment.
[ Current State & Action ]
│
├──► [ Process 1: Mean Dynamics ] ──► Predicted Next State
│
└──► [ Process 2: Noise Intensity ] ──► Predicted Variance (State-Dependent)
This localized understanding of environmental stochasticity prevents the model from washing out critical details. In interpretable dynamics models for data-efficient reinforcement learning, Carl Henrik Ek et al. show that learning a separate process for noise intensity prevents agents from over-generalizing stability, allowing them to distinguish clearly between regions of high turbulence and regions of predictable dynamics.
This concept can also be implemented using mixture models. David Wingate et al. introduced this via the Mixture of Predictive Linear Gaussian (MPLG) model in Mixtures of Predictive Linear Gaussian Models for Nonlinear, Stochastic Dynamical Systems, which assigns unique noise statistics to individual components within a mixture model. As the system transitions through different regions of the state space, different components become active, naturally allowing the modeled uncertainty to fluctuate. Even within formal logic frameworks, Fahiem Bacchus et al. showed in Reasoning about Noisy Sensors and Effectors in the Situation Calculus that noise parameters and likelihood functions can be defined via first-order logic, allowing sensor accuracy or movement variance to vary dynamically based on environmental conditions.
Why Differentiable Filters and Smoothers Require State-Dependent Noise
State estimation frameworks, such as Kalman filters or factor graphs, rely heavily on knowing when to trust sensor observations versus when to trust internal physics models. If a sensor becomes temporarily blocked, a homoscedastic filter will continue to trust the corrupted sensor data just as much as it did before, leading to catastrophic tracking failures.
By integrating deep learning with classical probabilistic structures, modern architectures can learn state-dependent noise models that dynamically adjust uncertainty based on environmental conditions. As Alina Kloss et al. demonstrate in How to Train Your Differentiable Filter, a differentiable filter can learn to spike its predicted observation noise during sensor occlusions, forcing the system to rely more heavily on its internal dynamics model.
This concept is pushed further into trajectory optimization by Brent Yi et al. in Differentiable Factor Graph Optimization for Learning Smoothers, who unroll nonlinear factor graph optimizers to allow end-to-end learning of these state-dependent uncertainties. When the reliability of a sensor changes dynamically, the unrolled optimizer propagates these localized uncertainties through time, ensuring the smoother accurately weights every piece of information.
In highly nonlinear, non-Gaussian settings, Arash Mehrjou et al. show in Deep Nonlinear Non-Gaussian Filtering for Dynamical Systems that implicit generative objectives allow models to automatically increase variance at sharp transition points or discontinuities. This provides a physically accurate representation of complex biological or physical processes where noise levels naturally fluctuate based on the latent state, a phenomenon also explored by Antonio Khalil Moretti et al. in Smoothing Nonlinear Variational Objectives with Sequential Monte Carlo.
Stabilizing Reinforcement Learning and Control
In Reinforcement Learning, heteroscedasticity is not just an environmental nuisance; it is an inherent property of the learning process itself. Because an agent's policy is constantly evolving, the distribution of targets and temporal difference (TD) errors changes drastically across different states, as noted by Nikolay Nikolov et al. in Information-Directed Exploration for Deep Reinforcement Learning.
Standard deep Q-learning algorithms minimize the Mean-Squared Bellman Error (MSBE), which implicitly assumes that the noise in function approximation is homoscedastic and normally distributed. However, David Yu-Tung Hui et al. show in Double Gumbel Q-Learning that modeling TD errors as state-dependent noise allows the learning objective to better accommodate varying levels of uncertainty across the state space.
Furthermore, assuming a fixed-variance Gaussian to model return distributions prevents an agent from distinguishing between inherent environmental noise (aleatoric uncertainty) and a lack of data (epistemic uncertainty). As Simon Schmitt et al. argue in Exploration via Epistemic Value Estimation, capturing state-dependent aleatoric uncertainty is essential for isolating true epistemic uncertainty, enabling highly efficient, targeted exploration instead of wasting time exploring states that are merely inherently noisy.
Total Uncertainty ──► Isolate State-Dependent Aleatoric Noise (Inherent)
└──► Reveal True Epistemic Uncertainty (Lack of Data) ──► Targeted Exploration
When noise is non-uniform, standard statistical estimation techniques often yield loose performance guarantees that fail to reflect the underlying variance of the system. To address this, Andrew Wagenmaker et al. introduced a robust estimation approach (FORCE) in First-Order Regret in Reinforcement Learning with Linear Function Approximation: A Robust Estimation Approach that handles state-and-time-varying fluctuations, scaling the agent's learning curve with the actual task difficulty rather than a conservative, worst-case scenario.
Safe Navigation and Policy Optimization
For an agent to plan safely, it must actively avoid regions of high risk. Models that assume uniform noise fail to identify these volatile zones, leading to suboptimal or dangerous control policies, as demonstrated by Stefan Depeweg et al. in Learning and Policy Search in Stochastic Dynamical Systems with Bayesian Neural Networks. In robotic control, where state-action pairs dictate physical safety, Rel Guzman Apaza et al. highlight in Adaptive Model Predictive Control by Learning Classifiers that ignoring heteroscedastic noise leads to inaccurate predictions and poor control performance.
By explicitly mapping state-dependent noise, control frameworks can achieve distributional robustness. For example, Rui Liu et al. developed a data-driven distributionally robust optimal control framework ($D^3ROC$) in Data-Driven Distributionally Robust Optimal Control with State-Dependent Noise that learns localized noise distributions (such as localized wind or sensor interference) directly from data. This allows the controller to naturally steer robots away from high-noise areas without requiring manual parameter tuning.
This state-correlated uncertainty awareness is equally vital in multi-agent settings. Nazım Kemal Üre et al. demonstrated in Experimental Demonstration of Multi-Agent Learning and Planning under Uncertainty for Persistent Missions with Automated Battery Management that accounting for state-correlated events, such as how fuel consumption or sensor failure probabilities depend on the agent's current state, significantly improves long-term mission performance and planning efficiency. Ultimately, accounting for state-dependent noise ensures that policy gradients and exploration strategies are computed using a simulator that faithfully reflects the non-uniform volatility of the real world.
Key Foundational Work
Mixtures of Predictive Linear Gaussian Models for Nonlinear, Stochastic Dynamical Systems (2006) — contributed 3 concept elements to this direction.
Reasoning about Noisy Sensors and Effectors in the Situation Calculus (1998) — contributed 1 concept elements to this direction.