TL;DR
This paper introduces Adversarial Missingness, a framework where an intelligent adversary permanently blocks specific state features to degrade a Reinforcement Learning (RL) agent's performance. To counter this, the authors develop an adversarial training algorithm that dynamically samples and masks critical features during training, significantly outperforming standard random-dropout baselines in complex environments.
What Problem Does This Paper Address?
When deploying RL agents in real-world scenarios—like robotics or autonomous driving—ensuring safety under adverse conditions is paramount. Traditional adversarial RL typically focuses on $\ell_\infty$-norm bounded perturbations, where an attacker injects tiny, imperceptible noise across all input features.
However, this threat model fails to capture common real-world faults, such as a camera being obstructed, a sensor completely breaking, or a malicious actor cutting off a specific data stream. While standard "RL with missing data" techniques address random sensor dropouts, they do not prepare an agent for an intelligent adversary who intentionally targets and exploits the most critical features. This paper bridges that gap by formalizing adversarial feature missingness and proposing a robust training solution.
Methods
The authors formalize the problem by modifying a standard Markov Decision Process (MDP) to include a missingness function $m_u(s)$, which replaces a subset of features $u$ with a masked value ($\bot$). Crucially, $u$ is globally fixed throughout an episode to accurately simulate permanent sensor failure.
To train an agent robust to this adversary, they propose the following pipeline:
- State Representation: Instead of using complex imputation models to guess the missing values, the masked features are simply set to zero and concatenated with a binary indicator vector that explicitly tells the agent which features are missing.
- Stochastic Adversary: Rather than training against a purely deterministic adversary (which always picks the single worst feature and causes training instability), they implement a stochastic adversary.
- Adversarial Score Update: The training environment maintains a probability distribution, $p_{miss}$, over which features to hide. Every $\zeta = 5000$ time steps, the agent's value function is evaluated under each possible missing feature to compute "adversarial scores." The distribution $p_{miss}$ is then updated using an inverse relationship: features that cause the sharpest drop in performance are sampled more frequently in subsequent training episodes.
The underlying RL policy is optimized using the standard Soft Actor-Critic (SAC) algorithm.
Datasets
The approach was evaluated on three standard continuous control benchmark environments from the MuJoCo physics engine:
- Hopper-v5: A low-dimensional task with 11 observation features and 3 actions.
- Walker2d-v5: A medium-dimensional task with 17 observation features and 6 actions.
- Ant-v5: A higher-dimensional task with 105 observation features and 8 actions.
Key Insights and Ideas
- The Curse of High-Dimensionality: The benefit of adversarial training scales dramatically with the size of the observation space. In low-dimensional environments like Hopper, random uniform feature dropout provides a decent proxy for robustness. However, in high-dimensional environments like Ant, random dropout completely fails because the agent develops severe "blind spots" that an adversary can easily exploit.
- Smooth Adversarial Updates Prevent Forgetting: Constantly exposing the agent to the absolute worst-case feature causes the policy to oscillate and forget how to handle previously mitigated missing features. A smooth, probability-based update rule stabilizes the competitive dynamic between the agent and the adversary.
Key Results
Performance was measured using the minimum episodic return across all possible single-feature missingness scenarios (representing the worst-case attack).
The adversarially trained agents achieved superior worst-case robustness across all three environments compared to a baseline trained on uniform random missingness. The most striking result occurred in the Ant-v5 environment: while the uniform baseline suffered a catastrophic performance collapse (dropping into negative reward territory), the adversarially trained agent successfully maintained high positive rewards.
Below are the training curves showing the minimum score under attack across the three environments:
Limitations and Future Work
- Single Feature Constraint ($k=1$): The current experimental evaluation is limited to scenarios where only one feature is missing at a time. Scaling the framework to handle multiple missing features ($k > 1$) introduces a combinatorial explosion in the number of possible feature subsets.
- Computational Overhead: Evaluating the agent's performance across every single feature combination to update $p_{miss}$ becomes computationally expensive as the state space grows.
- Future Directions: The authors propose exploring more efficient ways to estimate adversarial scores without exhaustive evaluation, potentially by leveraging feature attribution methods from Explainable AI (XAI).
Summary updated