Abstract
Bandit algorithms are widely used in sequential decision problems to maximize cumulative reward. One potential application is mobile health (mHealth), where personalized interventions are based on user-specific information. However, factors like data collection frequency and GPS monitoring can impact app performance and user adherence. To balance data utility with performance constraints, one must assess the usefulness of variables. Because bandit feedback data are sequentially correlated, traditional testing procedures for independent data do not apply. While statistical testing procedures exist for actor-critic bandit algorithms, they typically require a correctly specified reward model (critic). In practice, misspecification is frequent. We propose a modified actor-critic algorithm robust to critic misspecification and derive a novel testing procedure for the actor parameters.
Introduction
In contextual bandit problems, agents use "side information" (contexts) to choose actions that maximize rewards. In mHealth, this might involve using heart rate or location data to trigger health notifications. However, collecting this data incurs costs: battery drain, latency, and privacy concerns. Consequently, researchers need robust statistical inference to determine if a specific variable actually improves decision-making.
Existing Actor-Critic (AC) frameworks—where an Actor learns the policy and a Critic estimates the reward—rely on the assumption that the critic is "correctly specified." If the reward function is non-linear but modeled as linear, standard statistical tests often fail, leading to false conclusions about variable utility. This paper addresses this gap by introducing a framework that remains valid even under model misspecification.
The Actor-Improper Critic Framework
The proposed Actor-Improper Critic algorithm ensures statistical validity through three core innovations:
- Compatibility Condition: The algorithm utilizes a functional form for the critic that is "compatible" with the actor's softmax policy. This ensures the actor can converge to the optimal policy even if the critic is only an approximation of the true reward.
- Importance-Weighted RMS: To handle the sequential correlation of bandit data, the authors employ importance weighting. This adjusts the loss function based on action probabilities, effectively de-biasing the data.
- Asymptotic Normality: The authors mathematically prove that as time ($t$) increases, the actor's parameter estimates follow a multivariate normal distribution. This allows practitioners to use standard Z-tests to identify significant contextual variables.
Empirical Results
Statistical Power and Robustness
In synthetic experiments involving a misspecified non-linear reward function, the proposed method demonstrated superior robustness. While the standard Actor-Critic test failed to reject the null hypothesis for important variables in approximately 23% of cases, the robust method identified them with 99% accuracy. Both methods maintained low false-positive rates (7-9%) for "junk" variables that had no impact on rewards.
Real-World Performance and Stability
The algorithm was also tested on the Recovery Record Dataset, which tracks clinician communication strategies for patients with eating disorders. The proposed method achieved cumulative rewards comparable to the popular $\epsilon$-greedy approach but with significantly lower variance, indicating more stable and reliable performance in real-world settings.
Conclusion and Future Work
This work bridges the gap between theoretical regret minimization and practical statistical inference. By allowing for an "improper" critic, the algorithm enables researchers to identify and discard unnecessary variables, reducing the sensing burden in wearable technology. Future research will focus on scaling the non-convex optimization required for actor parameters to higher-dimensional spaces and developing sequential testing procedures to drop variables mid-trial.
Summary updated