CRTConjoint: Conditional Randomization Testing (CRT) Approach for Conjoint Analysis

Dae Woong Ham, Kosuke Imai, Lucas JansonOpenalex Unknown 2022 · DOI · OpenAlex

Abstract: Computes p-value according to the CRT using the HierNet test statistic. For more details, see Ham, Imai, Janson (2022) "Using Machine Learning to Test Causal Hypotheses in Conjoint Analysis" doi:10.48550/arXiv.2201.08343.

Introduction

Conjoint analysis is a cornerstone of social science and market research, utilized to measure preferences by asking respondents to choose between multi-attribute profiles. For instance, a respondent might choose between two job candidates with varying levels of education, experience, and country of origin. However, the validity of these experiments rests on several "hidden" assumptions that are often difficult to verify: the absence of carryover effects (where previous tasks influence current choices), fatigue effects (where respondent attention wanes), and profile order effects (where physical placement biases selection).

Traditional statistical methods often struggle to test these assumptions with high power, especially when interactions between attributes are complex. The CRTConjoint R package addresses this by providing a non-parametric testing suite that leverages modern machine learning to identify if these experimental "nuisances" are biasing results.

The Conditional Randomization Test (CRT) Framework

The package implements the Conditional Randomization Test (CRT). Unlike standard tests that rely on large-sample theory, the CRT generates a null distribution by repeatedly re-sampling the treatment (the conjoint factors) according to the known experimental design while keeping the outcomes and other covariates fixed.

Table of arguments for the CRTConjoint package, highlighting the flexibility in experimental design (Uniform vs. Manual).

The core of this implementation is the HierNet test statistic. HierNet is a specialized Lasso-based model that enforces a hierarchical structure: it only considers an interaction between two attributes if the main effects of those attributes are also significant. This is particularly well-suited for conjoint analysis, where researchers are often interested in how specific combinations of features drive choices. By using machine learning as a test statistic rather than just for prediction, the package can detect complex, non-linear signals that simple linear models might miss.

Key Functions and Experimental Artifacts

CRTConjoint provides four primary functions to test various causal hypotheses:

  • CRT_pval: Tests if a specific factor significantly influences the outcome.
  • CRT_carryovereffect: Tests if previous tasks influence current choices.
  • CRT_fatigueeffect: Tests if respondent behavior changes over the course of the experiment.
  • CRT_profileordereffect: Tests for bias based on the physical placement (left vs. right) of profiles.

For testing fatigue and carryover effects, the package requires a respondent index to track individual behavior across multiple tasks.

Function parameters for artifact testing, including the respondent index and computational speedup features.

Computational Considerations and Data

Because the CRT requires fitting a model hundreds of times (the default is $B=200$ resamples), the process is computationally intensive. To mitigate this, the package includes several optimizations:

  • Parallel Computing: Support for doSNOW and foreach allows the workload to be distributed across multiple cores.
  • Speedup and Tolerance: A speedup parameter employs computational tricks to accelerate the function, while tol (tolerance) settings help manage the optimization fit for HierNet.
  • C++ Integration: The use of Rcpp ensures that the underlying calculations are as efficient as possible.

The package includes the immigrationdata dataset, a subset of the study by Hainmueller et al. (2014), featuring 1,000 choices regarding immigrant candidates. This dataset serves as a benchmark for demonstrating how to test for fatigue and carryover effects in real-world contexts.

Conclusion

CRTConjoint offers a robust, machine-learning-enhanced framework for validating conjoint experiments. By providing standardized p-values and observed test statistics, it allows researchers to rigorously test for experimental artifacts without relying on restrictive parametric assumptions. While computationally demanding, its parallelization and optimization features make it a viable tool for modern social science research.

Summary updated

Abstract

Computes p-value according to the CRT using the HierNet test statistic. For more details, see Ham, Imai, Janson (2022) "Using Machine Learning to Test Causal Hypotheses in Conjoint Analysis" &lt;<a href="https://doi.org/10.48550/arXiv.2201.08343" target="_top">doi:10.48550/arXiv.2201.08343</a>&gt;.
BibTeX
@inproceedings{ham2022crtconjoint,
  author = {Dae Woong Ham and Kosuke Imai and Lucas Janson},
  title = {{CRTConjoint: Conditional Randomization Testing (CRT) Approach for Conjoint Analysis}},
  booktitle = {Openalex Unknown},
  year = {2022},
  doi = {10.32614/cran.package.crtconjoint},
}
Figure