Robust Tests in Online Decision-Making

Gi-Soo Kim, Jane P. Kim, Hyun-Joon Yang

Abstract

Bandit algorithms are widely used in sequential decision problems to maximize cumulative reward. One potential application is mobile health (mHealth), where personalized interventions are based on user-specific information. However, factors like data collection frequency and GPS monitoring can impact app performance and user adherence. To balance data utility with performance constraints, one must assess the usefulness of variables. Because bandit feedback data are sequentially correlated, traditional testing procedures for independent data do not apply. While statistical testing procedures exist for actor-critic bandit algorithms, they typically require a correctly specified reward model (critic). In practice, misspecification is frequent. We propose a modified actor-critic algorithm robust to critic misspecification and derive a novel testing procedure for the actor parameters.

Introduction

In contextual bandit problems, agents use "side information" (contexts) to choose actions that maximize rewards. In mHealth, this might involve using heart rate or location data to trigger health notifications. However, collecting this data incurs costs: battery drain, latency, and privacy concerns. Consequently, researchers need robust statistical inference to determine if a specific variable actually improves decision-making.

Existing Actor-Critic (AC) frameworks—where an Actor learns the policy and a Critic estimates the reward—rely on the assumption that the critic is "correctly specified." If the reward function is non-linear but modeled as linear, standard statistical tests often fail, leading to false conclusions about variable utility. This paper addresses this gap by introducing a framework that remains valid even under model misspecification.

The Actor-Improper Critic Framework

The proposed Actor-Improper Critic algorithm ensures statistical validity through three core innovations:

  1. Compatibility Condition: The algorithm utilizes a functional form for the critic that is "compatible" with the actor's softmax policy. This ensures the actor can converge to the optimal policy even if the critic is only an approximation of the true reward.
  2. Importance-Weighted RMS: To handle the sequential correlation of bandit data, the authors employ importance weighting. This adjusts the loss function based on action probabilities, effectively de-biasing the data.
  3. Asymptotic Normality: The authors mathematically prove that as time ($t$) increases, the actor's parameter estimates follow a multivariate normal distribution. This allows practitioners to use standard Z-tests to identify significant contextual variables.

Empirical Results

Statistical Power and Robustness

In synthetic experiments involving a misspecified non-linear reward function, the proposed method demonstrated superior robustness. While the standard Actor-Critic test failed to reject the null hypothesis for important variables in approximately 23% of cases, the robust method identified them with 99% accuracy. Both methods maintained low false-positive rates (7-9%) for "junk" variables that had no impact on rewards.

Statistical power and false positive rates for different parameters. The proposed method achieves 99% power for significant variables (θ1-θ3) compared to the standard AC.

Real-World Performance and Stability

The algorithm was also tested on the Recovery Record Dataset, which tracks clinician communication strategies for patients with eating disorders. The proposed method achieved cumulative rewards comparable to the popular $\epsilon$-greedy approach but with significantly lower variance, indicating more stable and reliable performance in real-world settings.

Performance comparison on the Recovery Record dataset. The proposed method shows comparable mean reward to ε-greedy but with significantly lower standard deviation (152.7 vs 274.9).

Conclusion and Future Work

This work bridges the gap between theoretical regret minimization and practical statistical inference. By allowing for an "improper" critic, the algorithm enables researchers to identify and discard unnecessary variables, reducing the sensing burden in wearable technology. Future research will focus on scaling the non-convex optimization required for actor parameters to higher-dimensional spaces and developing sequential testing procedures to drop variables mid-trial.

Summary updated

Abstract

Bandit algorithms are widely used in sequential decision problems to maximize the cumulative reward. One potential application is mobile health, where the goal is to promote the user's health through personalized interventions based on user specific information acquired through wearable devices. Important considerations include the type of, and frequency with which data is collected (e.g. GPS, or continuous monitoring), as such factors can severely impact app performance and users’ adherence. In order to balance the need to collect data that is useful with the constraint of impacting app performance, one needs to be able to assess the usefulness of variables. Bandit feedback data are sequentially correlated, so traditional testing procedures developed for independent data cannot apply. Recently, a statistical testing procedure was developed for the actor-critic bandit algorithm. An actor-critic algorithm maintains two separate models, one for the actor, the action selection policy, and the other for the critic, the reward model. The performance of the algorithm as well as the validity of the test are guaranteed only when the critic model is correctly specified. However, misspecification is frequent in practice due to incorrect functional form or missing covariates. In this work, we propose a modified actor-critic algorithm which is robust to critic misspecification and derive a novel testing procedure for the actor parameters in this case.
BibTeX
@inproceedings{kim2022robust,
  author = {Gi-Soo Kim and Jane P. Kim and Hyun-Joon Yang},
  title = {{Robust Tests in Online Decision-Making}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2022},
  url = {https://openreview.net/forum?id=sjxo1OmQ73},
}
Figure