Abstract
Many outcomes of interest in the social and health sciences, as well as in modern applications in computational social science and experimentation on social media platforms, are ordinal and do not have a meaningful scale. Causal analyses that leverage this type of data, termed ordinal non-numeric, require careful treatment, as much of the classical potential outcomes literature is concerned with estimation and hypothesis testing for outcomes whose relative magnitudes are well defined. Here, we propose a class of finite population causal estimands that depend on conditional distributions of the potential outcomes, and provide an interpretable summary of causal effects when no scale is available. We formulate a relaxation of the Fisherian sharp null hypothesis of constant effect that accommodates the scale-free nature of ordinal non-numeric data. We develop a Bayesian procedure to estimate the proposed causal estimands that leverages the rank likelihood. We illustrate these methods with an application to educational outcomes in the General Social Survey.
Introduction
In fields ranging from sociology to medicine, researchers frequently encounter data that is ordered but lacks a numerical scale. Examples include pain scales (none, mild, moderate, severe), educational attainment (high school, associates, bachelors, graduate), or Likert scales in survey research. These are "ordinal non-numeric" outcomes. While we know the order of the categories, we cannot say that the mathematical "distance" between a high school diploma and an associate degree is the same as the distance between a bachelor’s and a master’s degree.
The standard "gold standard" for measuring causal impact is the Average Treatment Effect (ATE). However, the ATE is mathematically undefined for non-numeric data because you cannot subtract or average categories. A common workaround is to assign integers (1, 2, 3...) to these categories, but this "cheating" assumes a linear scale that is often unjustifiable and can lead to misleading conclusions. This paper addresses the fundamental challenge of defining and estimating causal effects when the outcome scale is unknown.
A New Framework for Ordinal Causal Inference
The authors propose three major innovations to adapt the Rubin Causal Model for ordinal data:
1. Conditional Estimands
Instead of seeking a single global average, the framework focuses on conditional distributions. Specifically, it looks at the conditional median or mode. This allows researchers to ask more nuanced questions, such as: "For individuals who would have achieved a high school diploma under the control condition, what is their median outcome under the treatment?" This approach provides a granular view of how a treatment shifts specific segments of a population across categories.
2. Relaxing the Fisherian Sharp Null
Fisher’s classic "sharp null" hypothesis posits that a treatment has zero effect for every single unit. In numerical data, testing for a constant effect (e.g., everyone improves by 5 points) is straightforward. In ordinal data, "adding" an effect is impossible without a scale. The authors formulate a relaxed null hypothesis that accommodates scale-free data by testing whether an effect changes an outcome by more than a single category, providing a robust way to conduct hypothesis testing without arbitrary scaling.
3. Bayesian Rank Likelihood
To estimate these effects, the authors utilize a Bayesian approach leveraging rank likelihood. This method allows them to model an underlying "latent" continuous variable that drives the observed ordinal categories. Crucially, rank likelihood does not require the researcher to explicitly define the "cut-points" or thresholds between categories. This makes the model significantly more robust to different assumptions about how the latent variable maps to the observed categories.
The Latent Scale Trap and Separability
A key theoretical contribution of the paper is the warning against the "latent scale trap." It is tempting to assume a hidden continuous scale exists behind ordinal data, but the authors demonstrate that these models are often "unidentifiable." This means that different mathematical assumptions can produce the same observed data but lead to contradictory causal interpretations.
Furthermore, the authors prove (Proposition 4.1) that any causal estimand that cannot be expressed as separate functions of the treatment and control groups—what they call "non-separable" estimands—requires a model for the "science," or the joint distribution of potential outcomes. This highlights the necessity of the Bayesian framework they propose.
Application: Educational Attainment in the GSS
The authors applied their methodology to the 1994 General Social Survey (GSS), examining a sample of 835 male respondents. They investigated the causal effect of parental education (the "treatment," defined as at least one parent having a college degree) on the respondent's own educational attainment (the "outcome," categorized into five levels from less than high school to a graduate degree).
The results demonstrated that the effect of parental education is not uniform across the population:
- Significant Median Shift: For individuals who would have only attained a high school diploma if their parents were not college-educated, the "treatment" of having college-educated parents shifted their median outcome to at least a college degree.
- Robustness: The Bayesian rank likelihood approach showed that these conclusions were only "mildly sensitive" to the assumed correlation between potential outcomes, offering a more stable and defensible result than traditional integer-based methods.
- Hypothesis Testing: Using simulated data to validate their approach, the authors showed that their distance-based test statistic successfully rejected the null hypothesis of "no effect" with high confidence, where traditional methods might have failed to even define the test.
Limitations and Future Directions
While the framework provides a rigorous path forward, it is not without challenges. Causal inference always involves missing data (we never observe the same individual in both treatment and control states simultaneously). The authors acknowledge that their model depends on a correlation parameter ($\rho$) that is inherently unobservable. While they use heuristics like the Frechet-Hoeffding upper bound to address this, it remains a point where researcher judgment is necessary.
Additionally, the Bayesian MCMC (Markov Chain Monte Carlo) procedures required for rank likelihood are more computationally intensive than simple frequentist tests. Future work could expand this framework to include high-dimensional covariate adjustments, further refining the toolset for researchers working with complex, non-numeric social data.
Conclusion
By moving away from arbitrary numerical assignments and focusing on conditional distributions and rank-based estimation, this research provides a mathematically sound foundation for causal inference on ordinal data. It allows researchers to "measure what they can't count," providing clear, interpretable summaries of how treatments shift outcomes across ordered categories without imposing a false sense of linearity.
Summary updated