Abstract: Computes p-value according to the CRT using the HierNet test statistic. For more details, see Ham, Imai, Janson (2022) "Using Machine Learning to Test Causal Hypotheses in Conjoint Analysis" doi:10.48550/arXiv.2201.08343.
Introduction
Conjoint analysis is a cornerstone of social science and market research, utilized to measure preferences by asking respondents to choose between multi-attribute profiles. For instance, a respondent might choose between two job candidates with varying levels of education, experience, and country of origin. However, the validity of these experiments rests on several "hidden" assumptions that are often difficult to verify: the absence of carryover effects (where previous tasks influence current choices), fatigue effects (where respondent attention wanes), and profile order effects (where physical placement biases selection).
Traditional statistical methods often struggle to test these assumptions with high power, especially when interactions between attributes are complex. The CRTConjoint R package addresses this by providing a non-parametric testing suite that leverages modern machine learning to identify if these experimental "nuisances" are biasing results.
The Conditional Randomization Test (CRT) Framework
The package implements the Conditional Randomization Test (CRT). Unlike standard tests that rely on large-sample theory, the CRT generates a null distribution by repeatedly re-sampling the treatment (the conjoint factors) according to the known experimental design while keeping the outcomes and other covariates fixed.
The core of this implementation is the HierNet test statistic. HierNet is a specialized Lasso-based model that enforces a hierarchical structure: it only considers an interaction between two attributes if the main effects of those attributes are also significant. This is particularly well-suited for conjoint analysis, where researchers are often interested in how specific combinations of features drive choices. By using machine learning as a test statistic rather than just for prediction, the package can detect complex, non-linear signals that simple linear models might miss.
Key Functions and Experimental Artifacts
CRTConjoint provides four primary functions to test various causal hypotheses:
CRT_pval: Tests if a specific factor significantly influences the outcome.CRT_carryovereffect: Tests if previous tasks influence current choices.CRT_fatigueeffect: Tests if respondent behavior changes over the course of the experiment.CRT_profileordereffect: Tests for bias based on the physical placement (left vs. right) of profiles.
For testing fatigue and carryover effects, the package requires a respondent index to track individual behavior across multiple tasks.
Computational Considerations and Data
Because the CRT requires fitting a model hundreds of times (the default is $B=200$ resamples), the process is computationally intensive. To mitigate this, the package includes several optimizations:
- Parallel Computing: Support for
doSNOWandforeachallows the workload to be distributed across multiple cores. - Speedup and Tolerance: A
speedupparameter employs computational tricks to accelerate the function, whiletol(tolerance) settings help manage the optimization fit for HierNet. - C++ Integration: The use of
Rcppensures that the underlying calculations are as efficient as possible.
The package includes the immigrationdata dataset, a subset of the study by Hainmueller et al. (2014), featuring 1,000 choices regarding immigrant candidates. This dataset serves as a benchmark for demonstrating how to test for fatigue and carryover effects in real-world contexts.
Conclusion
CRTConjoint offers a robust, machine-learning-enhanced framework for validating conjoint experiments. By providing standardized p-values and observed test statistics, it allows researchers to rigorously test for experimental artifacts without relying on restrictive parametric assumptions. While computationally demanding, its parallelization and optimization features make it a viable tool for modern social science research.
Summary updated