Visualizing Statistical Mix Effects and Simpson's Paradox

Zan Armstrong, Martin WattenbergIEEE TVCG 2014 · DOI

Abstract

We discuss how "mix effects" can surprise users of visualizations and potentially lead them to incorrect conclusions. This statistical issue (also known as "omitted variable bias" or, in extreme cases, as "Simpson's paradox") is widespread and can affect any visualization in which the quantity of interest is an aggregated value such as a weighted sum or average. Our first contribution is to document how mix effects can be a serious issue for visualizations, and we analyze how mix effects can cause problems in a variety of popular visualization techniques, from bar charts to treemaps. Our second contribution is a new technique, the "comet chart," that is meant to ameliorate some of these issues.

Introduction: The Danger of Aggregation

In many business and policy contexts, decision-makers rely on weighted averages or sums—for instance, observing that "overall unemployment is down." However, these aggregates are influenced by two distinct factors: value changes (the metric itself changing within a group) and mix shifts (the relative size of the groups changing).

When a mix shift is dramatic, it can cause Simpson’s Paradox, where the aggregate moves in the opposite direction of every individual sub-segment. Standard visualizations like bar charts, treemaps, and maps often fail to show these two dimensions simultaneously. For example, a treemap comparing two time periods usually only shows the "end" state's sizes, subtly biasing the viewer toward groups that grew and hiding the "flow" of the population.

The Comet Chart: Unmasking Mix Effects

To address these challenges, we propose the Comet Chart, a specialized variation of a connected scatterplot. In this visualization, the x-axis represents the weight (e.g., size of the labor force) and the y-axis represents the value (e.g., unemployment rate).

Each sub-population is represented by a "comet"—a line segment connecting its starting state to its ending state. To show directionality, the segment is tapered: the "tail" is thin (start) and the "head" is thick (end), creating a sense of motion. A distinct aggregate comet (usually black) represents the overall population. If the sub-segment comets are all moving down (improving) but the aggregate comet is moving up (worsening), a mix effect is immediately visible.

Comet chart showing unemployment trends by education level (1992–2013). Note how the 'bachelors+' segment increases its share of the labor force while 'no hs' decreases.

The use of log-log scaling allows for the comparison of relative changes (slopes) regardless of the absolute starting size of the groups. The angle of the comet provides a quick heuristic for "combined metrics," such as total earnings.

Insights and Real-World Impact

The Comet Chart was tested using real-world economic datasets, including US Bureau of Labor Statistics (BLS) data and county-level unemployment rates. Through user testing with financial analysts at a Fortune 500 company, we found that the chart significantly impacted decision-making. One analyst noted that the chart revealed a mix shift so striking it "invalidated high-level goal setting," forcing the team to re-evaluate their targets.

The chart is particularly effective at outlier detection. In California's unemployment data, while the aggregate might suggest a general trend, the Comet Chart clearly identifies regional outliers like Imperial County, which behaves very differently from the massive Los Angeles labor market.

California county-level unemployment data (2012–2013) highlighting regional outliers like Imperial and Los Angeles.

Limitations and Future Directions

While powerful, the Comet Chart is intended for expert analysts rather than the general public due to its learning curve. It works best for 6 to 100 segments; beyond this, occlusion becomes a problem. Future work includes developing interactive filtering or clustering to handle higher-dimensional data and extending the model to show multi-point temporal paths without visual clutter. Additionally, adapting the "comet" metaphor for non-temporal comparisons—such as gender differences in university admissions—remains a promising research direction.

Summary updated

BibTeX
@article{armstrong2014visualizing,
  author = {Zan Armstrong and Martin Wattenberg},
  title = {{Visualizing Statistical Mix Effects and Simpson's Paradox}},
  journal = {IEEE Trans. Vis. Comput. Graph.},
  year = {2014},
  doi = {10.1109/tvcg.2014.2346297},
}
Figure