HOPE: Human-Centric Off-Policy Evaluation for E-Learning and Healthcare

Ge Gao, Song Ju, Markel Sanz Ausin, Min Chi

Introduction

Reinforcement Learning (RL) has the potential to revolutionize high-stakes human-centric domains such as healthcare and education by optimizing treatment plans and personalizing learning experiences. However, the deployment of unproven AI policies in these environments is often unethical or dangerous. Off-Policy Evaluation (OPE) provides a critical alternative, allowing researchers to estimate the performance of a new target policy using historical data collected from a different behavior policy.

In the paper "HOPE: Human-Centric Off-Policy Evaluation for E-Learning and Healthcare," researchers introduce a framework specifically designed to handle the complexities of human-centric data, which often lacks full observability and immediate feedback.

The Challenge: Partial Observability and Sparse Rewards

Standard OPE algorithms typically assume a fully observable environment where rewards are provided at every step. In real-world human-centric tasks, these assumptions rarely hold:

  1. Partial Observability: Internal human states, such as a student's cognitive load or a patient's underlying disease progression, are often hidden from the data.
  2. Sparse/Delayed Rewards: Feedback is usually only available at the end of a trajectory, such as a final exam score or a patient's discharge status.

Applying standard OPE to these scenarios often leads to inaccurate policy value estimates, which could result in the selection of suboptimal or even harmful strategies.

The HOPE Framework

HOPE (Human-centric OPE) addresses these challenges by reconstructing missing step-by-step rewards through a three-step process designed to mitigate the noise of sparse data and hidden states.

Overview of the HOPE framework, illustrating the transition from original sparse data to calibrated immediate rewards using nearest neighbors and critical observations.

1. Preliminary Reward Inference

Since only a final cumulative reward is typically available, HOPE first uses function approximation to infer how much of that total reward should be attributed to each individual step in the trajectory.

2. Identifying Critical Observations

Not all moments in a trajectory are equally significant. HOPE identifies "critical observations"—steps where the choice of action has a substantial impact on the final outcome. This is quantified by the maximum difference in Q-values at that state.

3. Nearest-Neighbor Calibration

To account for hidden human states, HOPE utilizes a K-Nearest Neighbors (KNN) approach. By looking at similar trajectories in the historical data, the framework refines the step-by-step rewards by averaging them across these "neighbors." This calibration focuses on the previously identified critical steps to avoid introducing noise from routine actions.

4. Policy Estimation

Finally, these reconstructed and calibrated rewards are integrated into a Weighted Importance Sampling (WIS) framework to estimate the total return of the target policy.

Experimental Validation and Results

The authors evaluated HOPE across three distinct environments: a synthetic sepsis simulator, real-world Electronic Health Records (EHR) from over 220,000 patient visits, and data from an Intelligent Tutoring System (ITS) involving over 1,300 students.

Key findings include:

  • Superior Policy Selection: In the ITS study, HOPE was the only method among nine state-of-the-art benchmarks to correctly identify the best-performing policy.
  • High Rank Correlation: On real-world medical data, HOPE achieved a Spearman’s rank correlation of 0.99, indicating an almost perfect alignment between predicted and actual policy rankings.
  • Statistical Alignment: Offline predictions made by HOPE regarding policy superiority were perfectly aligned with empirical results from live student trials.

Conclusion

HOPE demonstrates that reconstructing immediate rewards through neighbor-based calibration is a highly effective strategy for OPE in partially observable environments with sparse feedback. While future work may explore reducing variance for long-horizon tasks and adapting the framework for continuous actions, HOPE represents a significant step forward in safely evaluating AI for critical human-centric applications.

Summary updated

BibTeX
@inproceedings{gao2023hope,
  author = {Ge Gao and Song Ju and Markel Sanz Ausin and Min Chi},
  title = {{HOPE: Human-Centric Off-Policy Evaluation for E-Learning and Healthcare}},
  booktitle = {AAMAS},
  year = {2023},
}
Figure