Optimal Caliper Width for Propensity Score Matching of Three Treatment Groups: A Monte Carlo Study

Yongji Wang, Hong Cai, Chanjuan Li, Zhiwei Jiang, Ling Wang, Jiugang Song, Jielai XiaPLoS 2013 · DOI

Abstract

Propensity score matching is a method to reduce bias in non-randomized and observational studies. Propensity score matching is mainly applied to two treatment groups rather than multiple treatment groups, because some key issues affecting its application to multiple treatment groups remain unsolved, such as the matching distance, the assessment of balance in baseline variables, and the choice of optimal caliper width. The primary objective of this study was to compare propensity score matching methods using different calipers and to choose the optimal caliper width for use with three treatment groups. The authors used caliper widths from 0.1 to 0.8 of the pooled standard deviation of the logit of the propensity score, in increments of 0.1. The balance in baseline variables was assessed by standardized difference. The matching ratio, relative bias, and mean squared error (MSE) of the estimate between groups in different propensity score-matched samples were also reported. The results of Monte Carlo simulations indicate that matching using a caliper width of 0.2 of the pooled standard deviation of the logit of the propensity score affords superior performance in the estimation of treatment effects. This study provides practical solutions for the application of propensity score matching of three treatment groups.

Introduction

In observational research, the absence of random assignment often leads to selection bias, where baseline characteristics differ significantly between treatment groups. Propensity Score Matching (PSM) addresses this by pairing subjects with similar probabilities of receiving a treatment. While well-established for binary treatment comparisons, extending PSM to three or more groups—requiring the formation of "triplets"—presents unique challenges. A critical parameter is the caliper width: the maximum allowable difference in propensity scores between matched subjects. A narrow caliper reduces bias but sacrifices sample size (precision), while a wide caliper preserves sample size at the cost of increased bias.

Methodology

The researchers employed Monte Carlo simulations to identify the optimal caliper width for three-group matching.

  1. Distance Metric: They utilized a multi-group extension of the standard distance formula, calculating the average distance across all pairwise combinations of the logits of the three propensity scores.
  2. Matching Algorithm: A "greedy" matching algorithm was used to form 1:1:1 triplets without replacement.
  3. Simulation Design: 1,000 datasets were generated, each with 1,000 subjects and eight baseline variables (4 continuous, 4 dichotomous). Three pre-matching group size ratios (1:2:7, 1:2:3, and 2:3:5) were tested.
  4. Evaluation Metrics: Performance was assessed using Standardized Difference (balance), Matching Ratio (sample retention), Relative Bias (RB), and Mean Squared Error (MSE).

Results

The simulations revealed a clear trade-off between bias and variance. A caliper of 0.1 consistently minimized bias but resulted in low matching ratios. Conversely, a caliper of 0.8 maximized the matching ratio but failed to achieve adequate covariate balance.

The 0.2 caliper width (represented as 20% of the pooled standard deviation of the logit of the propensity score in the data) emerged as the optimal choice. It consistently yielded the lowest or near-lowest MSE, indicating the best overall accuracy in treatment effect estimation.

Performance metrics for the 1:2:3 ratio scenario, highlighting the optimal MSE at the 20% (0.2) caliper width.

In scenarios with 1:2:3 and 2:3:5 ratios, the 0.2 caliper reduced the standardized difference of all covariates to below 0.1, the standard threshold for negligible imbalance. The findings were further validated using real-world data from a heart failure study, where the 0.2 caliper achieved a 73.6% matching ratio and balanced 8 out of 9 major confounders.

Discussion and Conclusion

This study confirms that the "0.2 rule of thumb" commonly used in two-group PSM effectively extends to three-group scenarios. The MSE serves as a decisive metric, balancing the reduction of bias with the preservation of statistical power.

However, the study also highlights limitations. PSM becomes increasingly difficult as the number of treatment arms grows; beyond four groups, the "region of common support" often becomes too small for effective matching. Additionally, the computational intensity of forming triplets is significant. Future research should explore these dynamics for binary outcomes and more complex, correlated covariate structures.

In conclusion, for researchers conducting observational studies with three treatment groups, a caliper width of 0.2 of the pooled standard deviation of the logit of the propensity score is recommended as the optimal starting point for propensity score matching.

Summary updated

Abstract

Propensity score matching is a method to reduce bias in non-randomized and observational studies. Propensity score matching is mainly applied to two treatment groups rather than multiple treatment groups, because some key issues affecting its application to multiple treatment groups remain unsolved, such as the matching distance, the assessment of balance in baseline variables, and the choice of optimal caliper width. The primary objective of this study was to compare propensity score matching methods using different calipers and to choose the optimal caliper width for use with three treatment groups. The authors used caliper widths from 0.1 to 0.8 of the pooled standard deviation of the logit of the propensity score, in increments of 0.1. The balance in baseline variables was assessed by standardized difference. The matching ratio, relative bias, and mean squared error (MSE) of the estimate between groups in different propensity score-matched samples were also reported. The results of Monte Carlo simulations indicate that matching using a caliper width of 0.2 of the pooled standard deviation of the logit of the propensity score affords superior performance in the estimation of treatment effects. This study provides practical solutions for the application of propensity score matching of three treatment groups.
BibTeX
@inproceedings{wang2013optimal,
  author = {Yongji Wang and Hong Cai and Chanjuan Li and Zhiwei Jiang and Ling Wang and Jiugang Song and Jielai Xia},
  title = {{Optimal Caliper Width for Propensity Score Matching of Three Treatment Groups: A Monte Carlo Study}},
  booktitle = {PLoS},
  year = {2013},
  doi = {10.1371/journal.pone.0081045},
}
Figure