Abstract
In sequential decision-making environments, Actor-Critic bandit algorithms are frequently employed to optimize policies based on observed rewards. However, traditional statistical tests for these algorithms often rely on the assumption that the reward model (the "Critic") is correctly specified. In real-world applications like mobile health (mHealth), this assumption rarely holds. This paper presents a robust statistical testing procedure designed for scenarios where the Critic's model is improperly specified. By utilizing a centered advantage model and importance-weighted estimators, the proposed framework ensures valid inference and high statistical power even when the underlying mathematical assumptions about the environment are flawed.
Introduction
Sequential decision-making is a cornerstone of modern personalized interventions, such as mobile health apps that determine the optimal timing for notifications to encourage physical activity. To maximize rewards—such as user step counts—researchers utilize bandit algorithms. A critical component of deploying these systems is understanding which variables (features) truly drive performance.
The popular Actor-Critic framework addresses this by splitting the task: the "Actor" learns the optimal policy (the decision rule), while the "Critic" learns to predict the reward (the evaluation). Historically, performing statistical inference on the Actor's parameters required the Critic to be "correctly specified," meaning the mathematical model perfectly reflects the true environment. In practice, models are often misspecified—for example, assuming a linear relationship when the true dynamics are non-linear. When the Critic is wrong, standard statistical tests become invalid, leading to false discoveries or the failure to identify critical features.
Methodology
To address the risks of model misspecification, this research introduces the Actor-Improper Critic algorithm. This approach ensures that the Actor can still converge to an optimal policy and provide valid statistical tests even if the Critic's functional form is incorrect. The methodology relies on two primary innovations:
1. Centered Critic Model
Instead of attempting to model the absolute reward, the Critic is designed to model the "advantage." This is defined as the difference between the reward of a specific action and the average reward across all possible actions. By "centering" the model in this way, the algorithm satisfies a mathematical compatibility condition. This ensures that errors in the Critic's estimation of the absolute reward "cancel out" when the Actor evaluates the relative benefit of different actions.
2. Importance-Weighted Estimators
Data in bandit problems is collected "online," meaning the policy evolves as the agent learns. This creates a dependency between data points that violates the assumptions of standard independent and identically distributed (I.I.D.) statistical tests. The authors employ importance-sampling weights—the ratio of the current policy to the historical policy—to create unbiased reward estimates.
The authors prove that these parameter estimates follow an asymptotic normal distribution as the sample size increases. This theoretical foundation allows practitioners to use standard Z-tests to determine if a specific variable's coefficient is significantly different from zero, providing a rigorous method for feature selection.
Experimental Evaluation
The proposed method was evaluated using both controlled simulations and real-world data:
- Synthetic Data: The researchers created an environment where the true reward function was intentionally non-linear, while forcing the algorithm to use a linear model. This "misspecified" scenario tested the robustness of the inference procedure under duress.
- Recovery Record Dataset: This real-world mHealth dataset tracks patient adherence behaviors in eating disorder therapy. It involves complex interactions between clinicians and patients via a mobile app, offering a noisy and high-stakes environment for testing feature utility.
Key Results
The Actor-Improper Critic demonstrated significant improvements over traditional Actor-Critic methods in several dimensions:
- Statistical Power: In synthetic tests with misspecified models, the proposed method achieved a 99% rejection rate for non-zero parameters, correctly identifying useful variables. In contrast, standard Actor-Critic algorithms only achieved rejection rates between 64% and 77%, frequently failing to detect significant features.
- Variance Reduction: On the Recovery Record dataset, the method achieved cumulative rewards comparable to standard exploration strategies (like $\epsilon$-greedy) but with significantly lower variance in its parameter estimates, indicating more stable and reliable learning.
- Feature Identification: The algorithm successfully identified
last_msg(the number of days since the last clinician message) as a critical variable for predicting patient adherence. Simultaneously, it correctly identified that certain demographic variables were less relevant to the decision policy, allowing for more focused and efficient interventions.
Discussion and Limitations
While the Actor-Improper Critic provides a robust framework for inference, some challenges remain. The Actor's objective function is non-convex, requiring computationally intensive optimization methods like grid search followed by Nelder-Mead search. As the number of features grows, this complexity may become a bottleneck.
Furthermore, the current proof assumes that user contexts are independent over time. In many mHealth scenarios, a user's state is Markovian (today's state depends on yesterday's), a factor not yet accounted for in the current theoretical framework. Future work will also need to address the timing of these tests, moving toward sequential testing regimes that can provide insights throughout the duration of a trial rather than only at its conclusion.
Conclusion
The Actor-Improper Critic algorithm represents a significant step forward in making online decision-making algorithms more reliable for scientific discovery. By decoupling the validity of the Actor's inference from the perfection of the Critic's model, this framework allows researchers to confidently identify the drivers of behavior in complex, real-world environments where models are inevitably imperfect.
Summary updated