TL;DR
This paper introduces STREETWISE, a novel post-deployment policy shaping framework for offline Reinforcement Learning (RL) agents. It addresses the critical issue of performance degradation in sensitive applications like Real-Time Communication (RTC) when deployed systems encounter unforeseen, out-of-distribution (OOD) environmental changes. STREETWISE effectively adapts policies by characterizing and mitigating these disturbances in real-time, leading to significant improvements in robustness and performance.
What Problem Does This Paper Address?
Training RL agents directly in real-world, safety-critical systems like RTC is often infeasible due to cost and safety concerns. Offline RL offers a solution by learning from pre-collected datasets. However, a major challenge arises when these learned policies are deployed: they can fail dramatically if the real-world environment deviates from the data distribution they were trained on. These deviations, termed "exogenous stochastic disturbances," can alter the underlying transition dynamics of the system, leading to critical policy failures and poor generalization. The paper specifically targets this problem, aiming to make offline RL policies robust to these unseen, real-time environmental shifts, particularly in domains like Bandwidth Estimation (BWE) for RTC.
Methods
The core innovation of STREETWISE lies in its post-deployment policy shaping mechanism. Instead of trying to learn a universally robust policy during offline training (which is deemed impossible for unseen disturbances), STREETWISE modifies the behavior of an already trained offline RL policy after deployment.
The framework consists of three main modules:
- Offline RL Training: Standard model-free offline RL algorithms (like Implicit Q-Learning, IQL) are used to train an initial policy ($\pi$) and its associated value function ($Q$). This phase uses clean, disturbance-free data.
- OOD Detector: An LSTM Autoencoder (LSTM-AE) is trained on the offline trajectories. This autoencoder learns to reconstruct sequences of state-action-next-state transitions. When deployed, it takes real-time observations and its reconstruction error serves as a metric ($\rho_\epsilon$) to quantify the presence, position, span, and amplitude of OOD regions or disturbances in the environment.
- Post-Deployment Shaping: At runtime, the trained policy $\pi$ predicts an action. The OOD detector provides the disturbance metric $\rho_\epsilon$. STREETWISE then shapes the predicted action by perturbing it in the direction of the gradient of the value function ($\nabla_a Q(s, a)$), scaled by the disturbance metric $\rho_\epsilon$. This shaping potential ($\phi$) aims to steer the agent towards actions that are likely to yield higher returns even under the detected disturbances. The final action $a'$ is a combination of the original policy's action and this shaped perturbation.
Datasets
The paper uses two primary types of datasets:
- Microsoft's Bandwidth Estimation Challenge Dataset: This dataset consists of real-world trajectories collected from audio/video peer-to-peer (P2P) Microsoft Teams calls. It includes data from various network profiles, simulating different conditions like random loss, burst loss, fluctuating bandwidth, and stable bandwidth. This dataset is used for training and evaluating the BWE application.
- MuJoCo Environments (D4RL Benchmark): Standard locomotion tasks (HalfCheetah, Hopper, Walker2D) from the D4RL benchmark are used to evaluate the generalization capabilities of STREETWISE. For these experiments, custom noise is introduced by dynamically varying the viscosity of the environment during testing, simulating non-stationary dynamics.
Key Insights and Ideas
- The Impossibility of Universal Robustness: The paper argues that learning a policy that is robust to any unseen exogenous disturbance solely from offline data is practically impossible. This motivates the shift to a post-deployment adaptation strategy.
- Leveraging Value Function Extrapolation: While learned policies can be brittle in OOD regions, value functions ($Q$) tend to extrapolate better. STREETWISE capitalizes on this by using the gradient of the value function to guide action modifications.
- Real-time Disturbance Characterization: The LSTM Autoencoder acts as a crucial OOD detector, providing a quantifiable metric ($\rho_\epsilon$) that captures the nature of the environmental disturbance in real-time. This metric is key to adaptively shaping the policy.
- Shaping as a Complement, Not a Replacement: STREETWISE doesn't replace the learned offline policy but rather shapes its output. This preserves the learned knowledge while allowing for dynamic adaptation to unforeseen circumstances.
- Generalizability Beyond RTC: The framework's design, particularly the OOD detection and shaping mechanism, is presented as generalizable to other continuous control domains beyond BWE in RTC.
Key Results
- Significant Improvements in BWE: STREETWISE demonstrated substantial gains in video MOS (a key end-user metric) on various network profiles, showing up to an 18% improvement compared to state-of-the-art baselines in scenarios with random packet loss.
- Robustness to Burst Loss: Unlike some baselines (e.g., IQL+OPEX), STREETWISE maintained performance or improved it in the presence of fluctuating burst losses, indicating its resilience to complex disturbance patterns.
- Generalization to MuJoCo: Experiments on MuJoCo environments with introduced noise showed that STREETWISE consistently outperformed baselines like IQL, particularly when policies were trained on non-expert datasets.
- Stability on Stable Networks: On stable network profiles where the backbone offline policy was already optimal, STREETWISE's performance was never worse, ensuring safety and reliability.
Limitations and Future Work
- Stability Analysis: The paper acknowledges that deriving exact conditions for the stability of STREETWISE agents with respect to exogenous disturbances is a subject for future work.
- Non-Stationary Rewards: The current framework is primarily designed for disturbances in transition dynamics. The authors note that adding post-deployment support for disturbed reward functions is a future research direction.
- Computational Cost: While the paper prioritizes efficiency, the LSTM Autoencoder for OOD detection adds computational overhead during deployment.
- Regret Bounds and Guarantees: The authors suggest that providing regret bounds and formal guarantees for their method are important future research directions.
Summary updated