Regression Models for Ordinal Data

Peter McCullaghJournal of the Royal Statistical Society Series B: Statistical Methodology 1980 · DOI

TL;DR

Peter McCullagh’s 1980 seminal paper provides a robust statistical framework for regression when the outcome is ordinal—data that has a natural order (like "mild," "moderate," "severe") but no fixed numerical distance between categories. By introducing the proportional odds and proportional hazards models, McCullagh enables researchers to analyze these datasets without the pitfalls of treating categories as arbitrary numbers or ignoring their inherent sequence.

What Problem Does This Paper Address?

In fields ranging from sociology to medicine, researchers frequently encounter ordinal data. For example, a patient's recovery might be rated as "poor," "fair," or "good." Historically, statisticians handled this in two suboptimal ways:

  1. Treating it as nominal: Ignoring the order entirely, which throws away valuable information.
  2. Assigning arbitrary scores: Treating "poor" as 1, "fair" as 2, and "good" as 3. This assumes the "distance" between poor and fair is identical to the distance between fair and good—an assumption that is rarely justifiable.

McCullagh tackles the challenge of building a regression model that respects the stochastic ordering of the data—the idea that one group is generally "higher" on the scale than another—without assuming cardinality.

Methods

The paper proposes a general class of models based on cumulative probabilities. Instead of modeling the probability of being in a specific category, McCullagh models the probability of being in category $j$ or lower.

The core equation takes the form: $$link\{\gamma_j(x)\} = \theta_j - \beta^T x$$ Where:

  • $\gamma_j(x)$ is the cumulative probability for category $j$.
  • $\theta_j$ are "cut points" or thresholds that represent the boundaries between categories on an underlying latent continuous scale.
  • $\beta$ represents the effect of the covariates (explanatory variables).

The paper highlights two primary models:

  • The Proportional Odds Model: Uses the logit link function. It assumes that the odds of being below any given threshold are multiplied by a constant factor for each unit change in a covariate.
  • The Proportional Hazards Model: Uses the complementary log-log link function. This is particularly useful for asymmetric data or when the categories represent intervals of a survival-like process.

Estimation is performed using Maximum Likelihood (ML), and McCullagh demonstrates that the method of iteratively reweighted least squares converges reliably to the ML estimate.

Datasets

McCullagh illustrates his models using several classic datasets:

  • Tonsil Size: Data from 1398 children classified by tonsil size (not enlarged, enlarged, greatly enlarged) and whether they were carriers of Streptococcus pyogenes.
  • Family Income: Distribution of family income in the Northeast U.S. in 1960 vs. 1970 across seven categories.
  • Disturbed Dreams: A study of the severity of disturbed dreams in boys aged 5–15.
  • Vision Quality: A comparison of right-eye vision quality between men and women.
  • Taste-Testing: A five-treatment experiment where judges rated products from "Terrible" to "Excellent."

Key Insights and Ideas

  • Invariance Under Grouping: A major "aha moment" is that these models are invariant to how categories are grouped. If you merge "enlarged" and "greatly enlarged" tonsils into a single "enlarged" category, the estimate for the treatment effect ($\beta$) remains essentially the same. This makes the model robust to subjective categorization.
  • Latent Variable Interpretation: The models can be viewed as partitioning a hidden, continuous variable (like "severity" or "tolerance") into intervals. This provides a physical intuition for the "cut points" ($\theta_j$).
  • Stochastic Ordering: The models provide a formal way to say that one population is "stochastically larger" than another, meaning its cumulative distribution function is shifted.

Key Results

In the tonsil size example, the model clearly quantifies the impact of being a carrier.

Table 1: Tonsil size of carriers vs non-carriers

By analyzing the logits (as seen in the analysis below), McCullagh shows that the odds of having "greatly enlarged" tonsils are approximately 1.8 times higher for carriers than for non-carriers.

Table 2: Logit analysis of tonsil data

In the income distribution example, the proportional hazards model reveals a consistent shift toward higher income brackets from 1960 to 1970.

Table 5: Complementary log-log analysis of income data

The differences across categories in Table 5 are remarkably stable (ranging from 0.38 to 0.67), suggesting that the proportional hazards assumption is a good fit for describing how the entire income distribution shifted over the decade.

Limitations and Future Work

  • Sparse Data: The author notes that maximum likelihood estimates may fail to exist or become infinite if the data is too sparse (e.g., many categories with zero observations).
  • Link Selection: While logit and log-log links are preferred for their interpretability, choosing between them requires careful consideration of the data's symmetry.
  • Non-Linearity: The paper introduces extensions for non-linear models (where the "scale" or variance might also change between groups), but acknowledges these are computationally more demanding and harder to interpret.
  • Multivariate Responses: The framework primarily focuses on a single ordinal response; extending this to bivariate or multivariate ordinal responses is mentioned as an area for further research.

Summary updated

Abstract

Summary A general class of regression models for ordinal data is developed and discussed. These models utilize the ordinal nature of the data by describing various modes of stochastic ordering and this eliminates the need for assigning scores or otherwise assuming cardinality instead of ordinality. Two models in particular, the proportional odds and the proportional hazards models are likely to be most useful in practice because of the simplicity of their interpretation. These linear models are shown to be multivariate extensions of generalized linear models. Extensions to non-linear models are discussed and it is shown that even here the method of iteratively reweighted least squares converges to the maximum likelihood estimate, a property which greatly simplifies the necessary computation. Applications are discussed with the aid of examples.
BibTeX
@article{mccullagh1980regression,
  author = {Peter McCullagh},
  title = {{Regression Models for Ordinal Data}},
  journal = {Journal of the Royal Statistical Society Series B: Statistical Methodology},
  year = {1980},
  doi = {10.1111/j.2517-6161.1980.tb01109.x},
}
Figure