ABtree: An Algorithm for Subgroup-Based Treatment Assignment

Derek Feng, Xiaofei WangarXiv 2016 · DOI · arXiv

Abstract

Given two possible treatments, there may exist subgroups who benefit greater from one treatment than the other. This problem is relevant to the field of marketing, where treatments may correspond to different ways of selling a product. It is similarly relevant to the field of public policy, where treatments may correspond to specific government programs. And finally, personalized medicine is a field wholly devoted to understanding which subgroups of individuals will benefit from particular medical treatments. We present a computationally fast tree-based method, ABtree, for treatment effect differentiation. Unlike other methods, ABtree specifically produces decision rules for optimal treatment assignment on a per-individual basis. The treatment choices are selected for maximizing the overall occurrence of a desired binary outcome, conditional on a set of covariates. In this poster, we present the methodology on tree growth and pruning, and show performance results when applied to simulated data as well as real data.

Introduction

Standard A/B testing remains the gold standard for determining which of two options—be it website layouts, marketing strategies, or medical interventions—is superior on average. However, the concept of an "average" user is often a statistical abstraction. In reality, a marketing campaign might resonate with younger demographics while alienating older ones, or a drug might be highly effective only for patients with specific genetic markers. This phenomenon is known as treatment effect heterogeneity.

While many existing statistical methods can identify these subgroups, they often function primarily as exploratory tools. They may signal that groups differ but fail to provide a computationally efficient, actionable decision rule for assigning treatments to individuals to maximize a specific outcome. ABtree addresses this gap by framing treatment assignment as a direct optimization task: identifying the best treatment rule for every individual based on their unique characteristics (covariates).

Methodology

ABtree is built upon the foundation of Classification and Regression Trees (CART), but it replaces standard splitting criteria with a utility-focused objective function designed for decision-making.

The Objective Function

Instead of minimizing Gini impurity or entropy, which prioritize class purity, ABtree utilizes a custom function $Q(S_j)$. This function calculates the maximum possible expected profit (or success rate) within a subgroup by selecting the better-performing treatment for that specific node.

Splitting and Growth

The algorithm searches for a covariate split (e.g., "Is age > 18?") that maximizes the sum of the expected profits in the resulting child nodes. By doing so, it identifies the boundaries that most clearly differentiate treatment effectiveness. To ensure statistical reliability, the authors enforce a minimum number of observations for both treatments in every leaf node.

Pruning and Interpretability

To prevent overfitting, ABtree employs a "weakest-link" pruning strategy using a hold-out validation set to determine optimal tree depth. A significant advantage of this approach is its inherent interpretability; the final output is a decision tree that makes the logic behind every treatment assignment transparent—a critical requirement in high-stakes fields like medicine or public policy.

Experimental Evaluation

The performance of ABtree was validated using both synthetic and real-world datasets to test its robustness and accuracy.

Simulated Data

The authors generated 50 datasets, each containing 5,000 observations, using four distinct response functions ($\phi_k$). These functions were designed to model complex interactions between user traits and treatment outcomes. The data generation process followed a specific statistical framework to ensure the algorithm could be tested against varying levels of noise and interaction complexity.

Table 1: Simulation Framework for Data Generation

Real-World Data: National Supported Work (NSW)

The algorithm was also applied to the National Supported Work Demonstration dataset, a 1970s randomized experiment tracking whether job training (the treatment) increased the earnings of disadvantaged workers compared to a control group.

Results and Discussion

Performance Against Baselines

In all simulated scenarios, ABtree significantly outperformed both random treatment assignment and standard A/B testing. The study highlighted a major limitation of standard A/B testing: when treatment effects are highly heterogeneous (e.g., helping one group while hurting another equally), a standard test may report "no effect," missing the opportunity to optimize outcomes by segmenting the population.

Key Insights

  • Decision-First Focus: Unlike methods that focus on estimating the magnitude of an effect, ABtree prioritizes making the correct choice. It is more concerned with identifying the right treatment for the right person than with the precise statistical difference between options.
  • Efficiency and Noise Tolerance: ABtree proved robust against "noise" covariates—variables that are recorded but have no actual impact on treatment success. Furthermore, because it relies on simple counts and means, it is computationally faster than complex "black box" ensemble methods.
  • Pruning Observations: Interestingly, the authors found that pruning provided only marginal gains in their simulations, suggesting that for very large datasets, researchers might achieve high performance even without this computationally expensive step.

Real-World Findings

When applied to the NSW dataset, ABtree identified age as the primary driver of treatment success. The model suggested that individuals aged 18 or younger did not benefit from the program, whereas those over 18 saw significant benefits, particularly when further segmented by their previous year's earnings.

Conclusion and Future Work

ABtree provides a fast, interpretable, and effective method for subgroup-based treatment assignment. However, some limitations remain. Currently, the algorithm is optimized for binary outcomes (success/failure). While the logic extends to continuous data (such as total spend), formal methodology for such cases is a subject for future research. Additionally, evaluating these models on real-world data remains challenging due to the "counterfactual problem"—the impossibility of knowing what would have happened if an individual had received the alternative treatment. Future work will focus on refining performance proxies and extending the algorithm to more diverse data types.

Summary updated

Abstract

Given two possible treatments, there may exist subgroups who benefit greater from one treatment than the other. This problem is relevant to the field of marketing, where treatments may correspond to different ways of selling a product. It is similarly relevant to the field of public policy, where treatments may correspond to specific government programs. And finally, personalized medicine is a field wholly devoted to understanding which subgroups of individuals will benefit from particular medical treatments. We present a computationally fast tree-based method, ABtree, for treatment effect differentiation. Unlike other methods, ABtree specifically produces decision rules for optimal treatment assignment on a per-individual basis. The treatment choices are selected for maximizing the overall occurrence of a desired binary outcome, conditional on a set of covariates. In this poster, we present the methodology on tree growth and pruning, and show performance results when applied to simulated data as well as real data.
BibTeX
@misc{feng2016abtree,
  author = {Derek Feng and Xiaofei Wang},
  title = {{ABtree: An Algorithm for Subgroup-Based Treatment Assignment}},
  year = {2016},
  eprint = {1605.04262},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1605.04262},
}
Figure