Statistical Inference for Data Constrained to a Simplex

200 concepts129 papers1 proposal

In many scientific domains, the data we collect does not represent absolute quantities, but rather the relative proportions of a whole. Whether measuring the percentage of different minerals in a soil sample, the relative abundance of bacteria in the human gut, or the distribution of time spent on daily activities, these datasets share a fundamental mathematical property: their components are non-negative and must sum to a fixed constant (typically 1 or 100%).

This constraint places the data on a simplex, a restricted geometric space that defies the assumptions of standard Euclidean statistics. When researchers treat these proportions as independent variables, they encounter "spurious correlations" and biased estimates that can lead to false biological or physical conclusions.

The Trap of Spurious Correlation

The most intuitive way to understand the challenge of compositional data is through the lens of dependency. In a standard dataset, variable $A$ can increase without affecting variable $B$. However, on a simplex, the variables are intrinsically linked. As Narayanan et al. note, an increase in one proportion necessitates a decrease in others to maintain the sum-to-one constraint.

This mathematical necessity creates "spurious correlations." Pawlowsky-Glahn et al. demonstrate that two components can appear strongly negatively correlated simply because they share a common divisor, even if their underlying absolute concentrations are positively correlated. In microbiome research, Faust et al. show that this leads to "social networks" of microbes where interactions are artifacts of the data structure rather than biological reality.

To combat this, Quinn et al. propose using proportionality instead of correlation. By focusing on the ratios between components—which remain stable regardless of the total sum—researchers can identify associations that are "subcompositionally coherent," meaning the relationship between two variables doesn't change when a third variable is added or removed from the analysis.

The Geometry of the Simplex

Standard statistical tools like Principal Component Analysis (PCA) or linear regression assume that data exists in a flat, infinite Euclidean space. However, the simplex is a curved manifold. Li et al. explain that the simplex lacks the properties of a standard vector space, making operations like finding a "mean" or a "linear trend" mathematically ill-defined.

To bridge this gap, researchers often use log-ratio transformations. The most common, the Isometric Log-Ratio (ILR) transformation, maps the constrained simplex into an unconstrained Euclidean space. Andrews demonstrates how this allows for convex optimization in image segmentation, while Zhao et al. use it to improve the mapping of soil particles.

However, these transformations are not a panacea. Narayanan et al. found that in some deep learning tasks, a standard softmax layer—which naturally enforces the sum-to-one constraint—can outperform complex log-ratio adaptations. This suggests that while the geometry must be respected, the method of doing so depends heavily on the model architecture.

Beyond Gaussian Assumptions

Many machine learning models rely on Gaussian (normal) distributions to model noise or uncertainty. But as Leijon et al. argue in the context of speech coding, Gaussian models waste capacity by assigning probability mass to "impossible" values outside the simplex (e.g., negative frequencies). They propose Dirichlet Mixture Models as a more natural fit for bounded, ordered data.

This shift toward simplex-aware distributions is gaining traction in generative modeling. Zhang et al. introduce Dirichlet Diffusion, which replaces standard Gaussian noise with Dirichlet-based noise to better model data like color proportions. Similarly, Williams et al. develop bijections to map categorical distributions to Euclidean space for flow matching, ensuring that the generative process respects the unique ratios inherent in probability space.

Applications in Life Sciences and Beyond

The implications of compositional data analysis (CoDa) are particularly acute in the life sciences:

  • Microbiology: Yoon notes that quantitative absolute abundance data can reveal interactions that are completely masked in relative datasets.
  • Immunology: Yoo et al. provide a roadmap for analyzing immune cell proportions, showing how ignoring compositionality can lead to misleading conclusions about racial disparities in cancer patients.
  • Neuroscience: Zhu et al. apply CoDa to longitudinal brain development, uncovering how tissue proportions change in response to socioeconomic factors.

By moving away from the "Euclidean illusion" and adopting frameworks that respect the geometry of the simplex, researchers can ensure that their models reflect the physical and biological constraints of the systems they study. As Huang et al. conclude, evaluating model behavior within the mathematical simplex is the only way to ensure that feature importance and predictions remain consistent with the underlying data geometry.

Key Foundational Work

Modelling speech line spectral frequencies with dirichlet mixture models (2010) — contributed 3 concept elements to this direction.
History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 147 papers
  2. Version 1 · 2026-10-04 · 77 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_compositional_data_constraints_bbf04424,
  title = {compositional data constraints},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/compositional-data-constraints/rdr_bbf04424cbce428f95cbeb72b85b98c7},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 2, frozen text):

@misc{lacuna_compositional_data_constraints_bbf04424_v2,
  title = {compositional data constraints},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/compositional-data-constraints/rdr_bbf04424cbce428f95cbeb72b85b98c7/v/2},
  note = {Version 2, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake