Introduction
In real-world decision-making systems, such as e-commerce recommendation engines or streaming platforms, environments are rarely static. User preferences shift based on the time of day, day of the week, or seasonal trends. Traditional Off-Policy Evaluation (OPE) methods, which estimate the performance of a new policy using historical data, typically assume stationarity. They answer the question: "How would this policy have performed in the past?" However, for practitioners, the critical question is: "How will this policy perform in the future?"
This "future gap"—the lack of observed data for the target time period—leads to significant bias in conventional OPE. This paper introduces OPFV (Off-Policy Estimator for the Future Value), a framework designed to bridge this gap by leveraging recurring temporal structures.
Methodology: The OPFV Framework
The core innovation of OPFV is the recognition that while the future is unobserved, its structure often mirrors the past. For instance, user behavior on a future Tuesday is likely similar to behavior on previous Tuesdays.
Technical Components
- Reward Decomposition: The reward function is decomposed into a predictable "time feature effect" (e.g., seasonality) and a "residual effect" (noise or non-periodic shifts).
- Time-Feature Importance Weighting: Unlike standard importance sampling, OPFV prioritizes historical data points that share temporal features (e.g., day of week, holiday status) with the future target time.
- Residual Modeling: To account for environmental shifts not captured by seasonal features, a reward regressor is integrated to further minimize bias.
- Granularity Optimization: The performance of OPFV depends on the "granularity" of time features. Coarse features (e.g., "Year") may miss nuances, while overly fine features (e.g., "Minute") lead to high variance due to data sparsity. The authors provide a data-driven method to optimize this bias-variance tradeoff.
Future Off-Policy Learning (F-OPL)
The framework extends beyond evaluation to policy optimization. By incorporating the OPFV estimator into a policy-gradient method (OPFV-PG), developers can train policies specifically optimized for anticipated future conditions.
Empirical Evaluation
The researchers validated OPFV using synthetic environments and the KuaiRec Dataset, a large-scale recommendation log from the Kuaishou app known for its periodic engagement cycles.
Key Results
OPFV consistently outperformed standard baselines, including Inverse Propensity Scoring (IPS), Doubly Robust (DR) estimators, and the "Prognosticator" model.
- Tracking Non-Stationarity: As shown in Figure 5, OPFV is uniquely capable of tracking the fluctuations of policy values across different future "seasons," whereas stationary baselines provide a flat, inaccurate estimate.
- Robustness: OPFV remains effective even as the environment becomes more non-stationary, provided the time features remain informative.
- Policy Improvement: In learning tasks, OPFV-PG achieved significantly higher rewards in future test periods. On the KuaiRec dataset, it reached a mean value of 1.67, substantially higher than IPS-PG (1.02).
Conclusion and Future Work
OPFV represents a significant advancement in off-policy evaluation for dynamic environments. By exploiting recurring temporal patterns, it provides a more reliable estimate of future performance than methods assuming stationarity.
Current limitations include a dependency on the availability of relevant time-series features; the model may struggle with "black swan" events that have no historical precedent. Future research will focus on more sophisticated residual modeling and handling non-stationarity in context distributions more effectively.
Summary updated