Abstract
This paper introduces a principled framework for policy transfer learning, focusing on the adaptation of decision-making rules from a data-rich source domain to a target domain where outcomes are unobserved. By integrating causal inference with semiparametric efficiency theory, the authors develop a "doubly robust" estimator capable of identifying optimal policies despite significant differences in feature distributions (covariate shift). Empirical evaluations on both simulated and real-world datasets demonstrate that the proposed Semiparametric Efficient (SE) estimator significantly outperforms baseline methods in terms of reward estimation, policy error reduction, and welfare improvement.
Introduction
In many practical scenarios, such as healthcare or public policy, the goal is to determine the optimal action (policy) for a specific population. However, researchers often face a data scarcity problem: detailed outcome data may exist for one population (the source), but only baseline characteristics are available for the population of interest (the target).
While standard transfer learning addresses predictive tasks (e.g., forecasting outcomes), policy learning is inherently counterfactual. It requires estimating what would happen under various actions, even when those actions were not taken or outcomes were not recorded in the target environment. This challenge is exacerbated by covariate shift, where the distribution of features $P(X)$ differs between the source and target domains, rendering direct model transfers unreliable.
Methodology
The proposed framework utilizes the potential outcomes framework of causal inference to bridge the gap between domains. The approach consists of three core components:
- Identifiability: The authors establish conditions—specifically Unconfoundedness and Transportability—under which the expected reward in the target domain can be linked to source domain data.
- The Semiparametric Efficient (SE) Estimator: Central to the paper is the derivation of the Efficient Influence Function (EIF) for the reward. This leads to a doubly robust estimator that combines an outcome regression model (predicting outcomes from features) and a weighting model (incorporating propensity and sampling scores). The estimator remains unbiased if either the outcome model or the weighting model is correctly specified.
- Policy Optimization: The optimal policy is identified by maximizing the SE reward estimator over a defined hypothesis space of potential policies.
Theoretical analysis provides a Generalization Error Bound, ensuring that the learned policy converges toward the optimal "oracle" policy as the sample size increases.
Experimental Evaluation
The framework was tested using two primary data sources:
- Simulated Datasets: Controlled environments with 512 source and 2,048 target individuals, allowing for precise manipulation of covariate shift and treatment probabilities.
- Real-World Data (Communities and Crime): A dataset of 1,994 US community records. New Jersey data served as the source, while other states acted as the target, testing the policy's adaptability to varying regional demographics.
As shown in the figure above, the SE method maintains superior stability as the "distance" between source and target populations increases. Traditional Importance Sampling (IPW) methods often suffer from high variance and instability under significant shift, whereas the doubly robust SE framework leverages the outcome model to maintain accuracy.
Key Results
The SE method consistently outperformed baseline "Direct" and "IPW" approaches across multiple metrics:
- Reward Estimation: Improved estimated rewards by 26.66% over the baseline in real-world tests.
- Policy Error Reduction: Reduced the squared error relative to the oracle policy by 84.35%.
- Welfare Change: Achieved a 92.69% improvement in total welfare gain compared to the baseline.
The sensitivity analysis regarding treatment probability (shown above) highlights the robustness of the SE estimator. While baseline methods degrade rapidly as treatment assignments become more skewed, the SE method retains high performance, demonstrating its efficiency in data-constrained settings.
Discussion and Limitations
The primary strength of this work lies in its achievement of the semiparametric efficiency bound, ensuring the lowest possible asymptotic variance. However, certain limitations remain:
- Concept Shift: The current method assumes the underlying causal mechanism (the relationship between $X$ and $Y$) is transportable. Future work could address "concept shift" where this relationship changes.
- Hypothesis Space Constraints: The quality of the learned policy is limited by the chosen policy class. If the optimal policy is more complex than the hypothesis space allows, systematic errors will persist.
In conclusion, this framework provides a robust, theoretically grounded approach to policy adaptation, offering significant advantages for decision-making in heterogeneous environments.