Off-Policy Evaluation and Learning for the Future under Non-Stationarity

Tatsuhiro Shimizu, Kazuki Kawamura, Takanori Muroi, Yusuke Narita, Kei Tateno, Takuma Udagawa, Yuta Saito

Introduction

In real-world decision-making systems, such as e-commerce recommendation engines or streaming platforms, environments are rarely static. User preferences shift based on the time of day, day of the week, or seasonal trends. Traditional Off-Policy Evaluation (OPE) methods, which estimate the performance of a new policy using historical data, typically assume stationarity. They answer the question: "How would this policy have performed in the past?" However, for practitioners, the critical question is: "How will this policy perform in the future?"

This "future gap"—the lack of observed data for the target time period—leads to significant bias in conventional OPE. This paper introduces OPFV (Off-Policy Estimator for the Future Value), a framework designed to bridge this gap by leveraging recurring temporal structures.

The concept of Future OPE (F-OPE) vs. Conventional OPE. While conventional methods estimate past performance, F-OPE aims to predict performance at a future target time $t'$.

Methodology: The OPFV Framework

The core innovation of OPFV is the recognition that while the future is unobserved, its structure often mirrors the past. For instance, user behavior on a future Tuesday is likely similar to behavior on previous Tuesdays.

Technical Components

  1. Reward Decomposition: The reward function is decomposed into a predictable "time feature effect" (e.g., seasonality) and a "residual effect" (noise or non-periodic shifts).
  2. Time-Feature Importance Weighting: Unlike standard importance sampling, OPFV prioritizes historical data points that share temporal features (e.g., day of week, holiday status) with the future target time.
  3. Residual Modeling: To account for environmental shifts not captured by seasonal features, a reward regressor is integrated to further minimize bias.
  4. Granularity Optimization: The performance of OPFV depends on the "granularity" of time features. Coarse features (e.g., "Year") may miss nuances, while overly fine features (e.g., "Minute") lead to high variance due to data sparsity. The authors provide a data-driven method to optimize this bias-variance tradeoff.

Future Off-Policy Learning (F-OPL)

The framework extends beyond evaluation to policy optimization. By incorporating the OPFV estimator into a policy-gradient method (OPFV-PG), developers can train policies specifically optimized for anticipated future conditions.

Empirical Evaluation

The researchers validated OPFV using synthetic environments and the KuaiRec Dataset, a large-scale recommendation log from the Kuaishou app known for its periodic engagement cycles.

Key Results

OPFV consistently outperformed standard baselines, including Inverse Propensity Scoring (IPS), Doubly Robust (DR) estimators, and the "Prognosticator" model.

  • Tracking Non-Stationarity: As shown in Figure 5, OPFV is uniquely capable of tracking the fluctuations of policy values across different future "seasons," whereas stationary baselines provide a flat, inaccurate estimate.
  • Robustness: OPFV remains effective even as the environment becomes more non-stationary, provided the time features remain informative.
  • Policy Improvement: In learning tasks, OPFV-PG achieved significantly higher rewards in future test periods. On the KuaiRec dataset, it reached a mean value of 1.67, substantially higher than IPS-PG (1.02).
Comparison of policy value estimation and MSE. (i) OPFV closely tracks the true policy value fluctuations over time, unlike stationary baselines. (ii) MSE of OPFV remains low as the informativeness of time features increases.

Conclusion and Future Work

OPFV represents a significant advancement in off-policy evaluation for dynamic environments. By exploiting recurring temporal patterns, it provides a more reliable estimate of future performance than methods assuming stationarity.

Current limitations include a dependency on the availability of relevant time-series features; the model may struggle with "black swan" events that have no historical precedent. Future research will focus on more sophisticated residual modeling and handling non-stationarity in context distributions more effectively.

Summary updated

BibTeX
@inproceedings{shimizu2025off,
  author = {Tatsuhiro Shimizu and Kazuki Kawamura and Takanori Muroi and Yusuke Narita and Kei Tateno and Takuma Udagawa and Yuta Saito},
  title = {{Off-Policy Evaluation and Learning for the Future under Non-Stationarity}},
  booktitle = {ACM SIGKDD Conference on Knowledge Discovery and Data Mining},
  year = {2025},
  doi = {10.1145/3690624.3709237},
}
Figure