A critical assessment of matching-adjusted indirect comparisons in relation to target populations

Ziren Jiang, Jialing Liu, Demissie Alemayehu, Joseph C. Cappelleri, Devin Abrahami, Yong Chen, Haitao ChuResearch 2025 · DOI

TL;DR

The paper identifies a "MAIC paradox" where matching-adjusted indirect comparisons can yield contradictory results—concluding Drug A is better than Drug B, or vice versa—simply by switching which trial provides the individual-level data. This highlights a critical vulnerability in how health technology assessments (HTAs) compare drugs without head-to-head trials, emphasizing that "superiority" is often dependent on the specific target population being analyzed.

What Problem Does This Paper Address?

In the world of Health Technology Assessment (HTA), researchers often need to compare two drugs (Drug A and Drug B) that have never been tested against each other in a single trial. Instead, they might have Trial 1 (Drug A vs. Placebo) and Trial 2 (Drug B vs. Placebo).

To compare them, researchers use Matching-Adjusted Indirect Comparison (MAIC). MAIC takes Individual Participant Data (IPD) from one trial and reweights the patients so their characteristics (like age or race) match the summary Aggregate Data (AgD) of the other trial.

The challenge arises from effect modifiers: variables that change how well a treatment works (e.g., a drug might be twice as effective in men as in women). If Trial 1 and Trial 2 have different proportions of these modifiers, MAIC is supposed to "level the playing field." However, the authors demonstrate that if the target population isn't clearly defined, two different companies can use the same data to "prove" their own drug is superior.

Methods

The authors illustrate the MAIC paradox using a mathematical framework for an "anchored" indirect comparison (where both drugs are compared to a common placebo).

  1. Weighting: They calculate balancing weights for subjects in the IPD trial so that the weighted mean of a covariate (in this case, race) matches the mean reported in the AgD trial.
  2. Estimation: They use a logit link to estimate the marginal treatment effect (log-odds ratio of survival) using the weighted IPD data.
  3. Comparison: They subtract the effect of Drug B (from the AgD trial) from the weighted effect of Drug A (from the IPD trial) to find the relative effectiveness.
  4. The Swap: They then repeat the entire process, but swap the roles—treating Trial 2 as the IPD source and Trial 1 as the AgD source—to see if the conclusion holds.

Datasets

The paper uses a synthetic dataset designed to represent a realistic clinical scenario.

  • Covariate: Race (Black vs. non-Black) serves as the sole effect modifier.
  • Trial AC: High proportion of non-Black participants; Drug A is less effective here.
  • Trial BC: Predominantly Black participants; Drug B is less effective here.
  • Outcome: Survival rates (Y=0 for survival, Y=1 for death).

By using synthetic data, the authors can precisely control the "ground truth" of treatment effects to demonstrate how the MAIC machinery produces the paradox.

Key Insights and Ideas

  • The MAIC Paradox: The most striking finding is that the "winner" of a drug comparison can flip depending on which trial's population is used as the target. If you match Trial A to Trial B's population, Drug A wins. If you match Trial B to Trial A's population, Drug B wins.
  • Population Sensitivity: MAIC results are only valid for the specific population represented by the AgD trial. There is no "universal" relative effectiveness when effect modifiers are imbalanced and have different magnitudes of modification across treatments.
  • The "Shared Effect Modifier" Assumption: For an indirect comparison to be truly valid across different populations, one must assume that the magnitude of the effect modification is the same for all treatments—an assumption that is often statistically untestable in a two-trial scenario.
  • Policy Implications: The authors argue that HTA agencies (like NICE in the UK) must demand a clear definition of the "target population" for the actual healthcare system, rather than just accepting a comparison to whatever trial population happens to be available as aggregate data.

Key Results

Using their synthetic example, the authors calculated the following:

  • Scenario 1 (Target = Trial BC population): Drug A was found to be significantly better than Drug B, with a log-odds ratio (logOR) of 0.425 (95% CI: 0.048, 0.802).
  • Scenario 2 (Target = Trial AC population): Drug B was found to be significantly better than Drug A, with a logOR of -0.402 (95% CI: -0.014, -0.790).

Both conclusions are mathematically "correct" within their respective contexts, yet they offer diametrically opposed guidance for clinical practice.

Limitations and Future Work

  • Moment Matching vs. Distribution Matching: MAIC typically only matches "moments" (like the mean). It doesn't account for the full distribution of covariates, which can lead to "ecological fallacy" issues.
  • Data Access: The primary limitation in solving this paradox is the lack of shared IPD. If HTA agencies had access to IPD from both trials, they could balance the entire covariate distribution and estimate effects for a specific, relevant national population.
  • Future Directions: The authors recommend the use of federated learning or secure data-sharing infrastructures. These would allow agencies to perform more robust analyses (like multilevel network meta-regression) without requiring companies to publicly release sensitive individual patient data.

Summary updated

Abstract

Matching-adjusted indirect comparison (MAIC) has been increasingly applied in health technology assessments (HTA). By reweighting subjects from a trial with individual participant data (IPD) to match the summary statistics of covariates in another trial with aggregate data (AgD), MAIC enables a comparison of the interventions for the AgD trial population. However, when there are imbalances in effect modifiers with different magnitudes of modification across treatments, contradictory conclusions may arise if MAIC is performed with the IPD and AgD swapped between trials. This can lead to the "MAIC paradox," where different entities reach opposing conclusions about which treatment is more effective, despite analyzing the same data. In this paper, we use synthetic data to illustrate this paradox and emphasize the importance of clearly defining the target population in HTA submissions. Additionally, we recommend making de-identified IPD available to HTA agencies, enabling further indirect comparisons that better reflect the overall population represented by both IPD and AgD trials, as well as other relevant target populations for policy decisions. This would help ensure more accurate and consistent assessments of comparative effectiveness.
BibTeX
@inproceedings{jiang2025critical,
  author = {Ziren Jiang and Jialing Liu and Demissie Alemayehu and Joseph C. Cappelleri and Devin Abrahami and Yong Chen and Haitao Chu},
  title = {{A critical assessment of matching-adjusted indirect comparisons in relation to target populations}},
  booktitle = {Research},
  year = {2025},
  doi = {10.1017/rsm.2025.10},
}
Figure