Assessments and developments in constructing a National Health Index for policy making, in the United Kingdom

Anna Freni Sterrantino, Thomas P. Prescott, Greg Ceely, Myer Glickman, Chris HolmesarXiv 2022 · DOI · arXiv

TL;DR

This paper provides a rigorous statistical "audit" of the UK’s first comprehensive National Health Index, evaluating how methodological choices—like how data is normalized or weighted—impact the final rankings of local authorities. The authors propose a more robust framework using winsorization and optimized weighting to ensure the index remains a reliable, transparent tool for policy-making.

What Problem Does This Paper Address?

Creating a single number to represent the "health of a nation" is a massive undertaking. The UK’s Office for National Statistics (ONS) developed the Health Index (HI) to track health across three domains: People (health outcomes), Lives (behaviors and circumstances), and Places (environmental factors).

However, composite indicators are notoriously sensitive. If you change the way you handle outliers or slightly adjust the weight of a specific indicator (like "access to green space"), the ranking of a city could jump or dive significantly. The authors address the "black box" problem of index construction, seeking to ensure that the HI is statistically coherent, conceptually sound, and resistant to arbitrary mathematical artifacts that could mislead policymakers.

Methods

The authors evaluate the "beta" version of the ONS Health Index (2015–2018) and propose a "modified" version. Their approach involves several key technical steps:

  • Data Treatment: They suggest moving from complex log-transformations to Winsorization (capping extreme outliers). This preserves the original scale of the data better and makes the final aggregation easier to interpret.
  • Weighting Systems: They compare the standard Factor Analysis (FA) weights used by the ONS against Principal Component Analysis (PCA) and a novel Optimized Weighting system. The latter uses an optimization algorithm to ensure that the weights assigned to domains actually reflect their intended "importance" rather than just their statistical variance.
  • Sensitivity and Uncertainty Analysis (UA): Using a Monte Carlo approach with 10,000 iterations, the authors tested how different combinations of normalization (Z-scores vs. Min-Max) and weighting affected the final rankings.
  • Aggregation: They explore the shift from arithmetic means (which allow a high score in one area to "compensate" for a low score elsewhere) to geometric means, which penalize uneven performance across domains.

Datasets

The study utilizes the England Health Index dataset (2015–2018).

  • Scope: 149 Upper Tier Local Authorities (UTLAs) in England.
  • Structure: 58 individual indicators nested within 17 subdomains, which are further nested within 3 main domains (People, Lives, Places).
  • Indicators: Range from clinical data (cancer prevalence, mortality) to socio-environmental data (housing affordability, air pollution, and even "distance to the nearest pharmacy").

Key Insights and Ideas

  • The "Wild Card" of Places: The "Healthy Places" domain often negatively correlates with "Healthy People" and "Healthy Lives." For example, a wealthy London borough might have excellent health outcomes (People) but terrible air quality and high crime (Places). This "trade-off" means that a single aggregate score can mask critical local failures.
  • The Middle-Rank Muddle: The sensitivity analysis revealed that while the "best" and "worst" performing areas stay relatively stable regardless of the math used, the middle 50% of local authorities are highly volatile. Small changes in weighting can cause these areas to swing wildly in the rankings.
  • Importance vs. Trade-off: In linear aggregation, weights are mathematically "trade-off ratios," but policymakers often interpret them as "importance coefficients." The authors highlight that if you want a weight to represent importance, you must account for the underlying correlation structure of the data.
  • Normalization is the Driver: At the domain level, the choice of how to normalize data (e.g., Z-score vs. Min-Max) was the single largest source of uncertainty in the final results.

Key Results

  • Ranking Shifts: In the authors' modified version of the index, 52% of local authorities shifted by fewer than 10 positions, but 9% shifted by more than 31 positions, demonstrating how much the "math under the hood" matters for local policy.
  • Weight Stability: PCA-derived weights were found to be remarkably stable over the four-year period, suggesting they are a robust alternative to the more complex Factor Analysis.
  • Domain Impact: The "Healthy Lives" domain showed the strongest influence on the overall index, confirming that behavioral and social determinants are the primary drivers of the UK's current health landscape.

Limitations and Future Work

The authors acknowledge that their analysis does not fully explore spatial-temporal correlations—how health in one area affects its neighbors over time. Furthermore, the index currently only covers England; expanding it to include Scotland, Wales, and Northern Ireland remains a challenge due to differences in data collection across the devolved nations.

Future work will focus on non-compensatory weighting. Policymakers are increasingly interested in models where a "perfect" score in air quality cannot simply cancel out a "failing" score in infant mortality, forcing a more holistic view of community well-being.

Summary updated

Abstract

Composite indicators are a useful tool to summarize, measure and compare changes among different communities. The UK Office for National Statistics has created an annual England Health Index (starting from 2015) comprised of three main health domains - lives, places and people - to monitor health measures, over time and across different geographical areas (149 Upper Tier Level Authorities, 9 regions and an overall national index) and to evaluate the health of the nation. The composite indicator is defined as a weighted average (linear combination) of indicators within subdomains, subdomains within domains, and domains within the overall index. The Health Index was designed to be comparable over time, geographically harmonized and to serve as a tool for policy implementation and assessment. We evaluated the steps taken in the construction, reviewing the conceptual coherence and statistical requirements on Health Index data for 2015-2018. To assess these, we have focused on three main steps: correlation analysis at different index levels; comparison of the implemented weights derived from factor analysis with two alternative weights from principal components analysis and optimized system weights; a sensitivity and uncertainty analysis to assess to what extent rankings depend on the selected set of methodological choices. Based on the results, we have highlighted features that have improved statistical requirements of the forthcoming UK Health Index.
BibTeX
@misc{sterrantino2022assessments,
  author = {Anna Freni Sterrantino and Thomas P. Prescott and Greg Ceely and Myer Glickman and Chris Holmes},
  title = {{Assessments and developments in constructing a National Health Index for policy making, in the United Kingdom}},
  year = {2022},
  eprint = {2210.05154},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2210.05154},
}

Loading related content...

Figure