Is Critical Mass Real? Testing Diversity Tipping Points Under Noisy Metadata and Hidden Confounding

The "critical mass" hypothesis holds that minority groups in a board, discipline, or conference committee have little influence until they reach roughly 20–30% of members, and that past this point outcomes change abruptly. The evidence for these thresholds is observational. The group's share is usually inferred by guessing gender from first names in bibliographic or corporate records, and the units that cross the threshold may differ in unrecorded ways from those that do not. Both problems can produce a tipping point that is not real, or hide one that is. Name-guessing errors blur a true step into a smooth curve. Errors whose rate varies by era or region can make a smooth trend look stepped. Struggling institutions that appoint women in a crisis can create a jump with no causal content. We propose a method that reports how much hidden confounding, and how much metadata error, a claimed threshold can survive before the data become equally consistent with no threshold at all.

The opportunity is that two literatures now supply the needed pieces without having met. Causal sensitivity analysis can produce the tightest possible bounds on a dose-response curve, meaning the outcome at each value of a continuous treatment, under a bounded amount of unmeasured confounding (Sharp Bounds for Generalized Causal Sensitivity Analysis). Metadata audits show that scholarly records are corrupted unevenly across time and place (One in Eight OpenAlex Abstracts Has Integrity Issues). Combining them turns "is there a 30% threshold?" into a question with a quantitative robustness answer. It also lets us ask a sharper version of the context-dependence question: do thresholds really differ across countries and institution types, or do the error rates of the tools used to measure them differ?

Core Claim

The quantity to report is a threshold robustness value. Let $s$ be a unit's true minority share and $\mu(s)$ its expected outcome. Let $[\mu^-_{\Gamma,\theta}(s), \mu^+_{\Gamma,\theta}(s)]$ be the sharpest bounds on $\mu(s)$ when unmeasured confounding is limited to strength $\Gamma \ge 1$, following the marginal sensitivity model for continuous treatments, and when metadata misclassification rates lie in a set $\theta$ calibrated from an audit. We define

$$ \Gamma^*(\theta) = \min \left\{ \Gamma : \exists f \in \mathcal{F}_{\text{smooth}} \text{ with } \mu^-_{\Gamma,\theta}(s) \le f(s) \le \mu^+_{\Gamma,\theta}(s) \ \forall s \right\}. $$

Here $\mathcal{F}_{\text{smooth}}$ is the class of curves with no jump and no change in slope larger than a pre-registered size $\delta$ anywhere in the support. A threshold claim is worth taking seriously only if $\Gamma^*$ is large under realistic $\theta$.

We make two predictions, both falsifiable. First, some published-style thresholds will have a small $\Gamma^*$ once name-classification error is modeled. Second, a measurable share of the apparent cross-context variation in threshold location will shrink once each context's own misclassification rates are used, because name-based gender inference fails unevenly across naming conventions.

Motivation and Positioning

Three lines of evidence motivate the question. The first is corporate governance. Fan et al. find that the link between board gender diversity and performance strengthens only past a critical mass. Board Gender Diversity and Firm Performance: Evidence from China finds a robust positive effect. In contrast, work on Malaysian small-cap firms (Board independence, women directors and firm performance in Malaysian small-cap firms) finds null or negative effects from token appointments and points to glass-cliff selection: women appointed to boards of already distressed firms.

The second is bibliometrics. Diversifying history ties women reaching about 30% of history PhDs in the mid-1980s to an expansion of research topics. The third is conference leadership. Gender Diversity of Conference Leadership reports step-like growth from 4% to 20% of leadership roles and a 0.75 correlation between program-chair and plenary-speaker diversity across 11 robotics conferences.

These studies share four features. The threshold location differs (20% versus 30%). Each treats name-guessed or photo-perceived gender as ground truth. None bounds hidden confounding. All acknowledge confounders such as broader cultural shifts or institutional distress.

Context-dependence in policy evaluation supplies the second ingredient. Uniform policies such as China's emissions trading pilots (Does an Emissions Trading Policy Improve Environmental Efficiency?) and its environmental tax reform (Identifying the Impact of Climate Policy on Urban Carbon Emissions) have effects that depend on regional baselines. That literature has learned to ask whether apparent heterogeneity reflects real mechanisms or differences in local measurement and enforcement. We borrow that question rather than its panels. A diversity mandate, or any rule pushing representation upward, is only as useful as the threshold it targets. If thresholds vary by context, the variation needs to survive measurement scrutiny.

The methods come from causal sensitivity analysis. The marginal sensitivity model with sharp, fast-to-compute bounds (Sharp Bounds for Generalized Causal Sensitivity Analysis) handles continuous treatments. It has been used to show policy effects robust up to $\Gamma<6$, but always for an average effect, never for the shape of a dose-response curve. Auditing Fairness under Unobserved Confounding shows how to make a sensitivity parameter interpretable by comparing it with the strength of observed covariates. Its disparity was robust to $\gamma = 1.12$, which corresponds to a confounder stronger than diabetes. We reuse this calibration so that $\Gamma^*$ can be read as "a confounder as strong as observed pre-period firm distress."

Two methods bound what we attempt. Causal Inference with Conditional Instruments Using Deep Generative Models needs thousands of samples to separate instruments from confounders, and most institution-level panels lack them. That pushes us toward partial identification rather than learned instruments. CausalPFN assumes no unmeasured confounding, which is exactly the assumption in dispute here.

Research Program

The work has four components.

Measurement model. We fit an explicit model linking observed shares to true shares. For a unit whose authors or members are gender-classified by name, the observed share is $\tilde{s} = (1-p)s + q(1-s)$ among classified members. Here $p$ is the rate at which women are labeled men and $q$ the reverse. A further term covers names the classifier refuses to label, which are dropped. We estimate $(p, q)$ and the drop rate separately for each naming region and era from a stratified, hand-labeled audit sample. Labels come from self-descriptions on institutional or ORCID-linked pages, using a human-consensus protocol like the one the OpenAlex audit used to validate its labels. That audit reports 96.0% agreement between its calibrated classifier and human consensus. The audit yields a credible set $\theta$, not a point estimate. Author fragmentation and merging also move member counts, so we propagate author-identity uncertainty using pairwise probabilities from a disambiguation model trained on S2AND. S2AND documents that disambiguation is worst for recent papers and junior researchers, precisely the people whose arrival defines a threshold crossing.

Bounding procedure. We compute the sensitivity bounds on $\mu(s)$ over a grid of true shares, integrating over the measurement model by importance weighting against $\theta$. We then solve the feasibility problem in the display equation as a convex program over piecewise-linear $f$ with slope-change penalties. This yields $\Gamma^*$ and, for any $\Gamma < \Gamma^*$, a confidence set for the threshold location.

Calibration of $\Gamma$. We compare $\Gamma$ with the confounding strength implied by leaving out observed covariates in turn: pre-period performance, institutional prestige, and field growth. This follows the benchmarking logic of Auditing Fairness under Unobserved Confounding.

Context analysis. We estimate threshold locations within contexts (world region, institution type, field) both before and after context-specific measurement correction. We then report the fraction of between-context variance in threshold location that the correction removes.

Data

The main real-data setting is OpenAlex, using institution-by-field-by-year units of authorship. The treatment is the share of women among active authors. The outcome is topical diversification of the unit's output, measured as the divergence of its topic distribution from the field's prior-decade distribution, in the spirit of Diversifying history. OpenAlex suits this question because its errors are documented to be non-random: abstract-integrity failures run at 21.1% before 2000 and about 6% for 2020–2025. We therefore restrict outcome construction to records passing an integrity filter and treat the filter rate as a covariate.

The secondary setting is conference leadership, rebuilt from public programs following Gender Diversity of Conference Leadership. The outcome is plenary-speaker diversity and the treatment is the share of women among program chairs. This setting is small, so it tests whether the method is informative at low sample sizes.

Baselines

The naive reference point is a two-way fixed-effects regression of the outcome on the observed share, linear in share and taken at face value. It shows what a standard analysis would conclude. Beating it here does not mean higher accuracy. It means the naive model reports a confident linear effect where our procedure shows the curve is unidentified, or misses a real step that we recover.

The strongest existing competitor for locating a threshold is a fixed-effects panel threshold regression: a piecewise model with an estimated breakpoint, fit on the same units with the same covariates and the same cross-validated tuning budget. It addresses non-linearity directly and is what an applied economist would run. We found no reported result on Lacuna for threshold-location recovery by such a method under metadata noise, so we establish its performance ourselves in the semi-synthetic tests. To beat it, our confidence set for the threshold must achieve nominal coverage of the planted threshold where its confidence interval undercovers, at comparable width.

The strongest existing competitor for hidden confounding is the marginal sensitivity model of Sharp Bounds for Generalized Causal Sensitivity Analysis, applied to a binary treatment "above 30%." This is the natural off-the-shelf use. It reports robustness of an average effect: its reported application stayed significant for $\Gamma<6$, with bounds computed in 0.4 seconds versus 137 for grid search. It cannot test whether a threshold exists, because dichotomizing fixes the threshold location in advance. We run it with the same flow-based density estimator and data as our procedure.

A network-proxy competitor follows Using Embeddings to Correct for Unobserved Confounding in Networks. We adjust for co-authorship-network embeddings trained jointly on treatment and outcome, then fit threshold regression. In its semi-synthetic test this approach moved a confounded estimate from 4.51 to 2.03 against a truth of 1.0. That shows it helps but leaves substantial bias, and it assumes a correctly observed network with no spillover between co-authors.

Two ablations isolate our key ingredients. One sets $\theta$ to zero error, giving sensitivity bounds without the measurement model. The other sets $\Gamma = 1$, giving measurement correction without the confounding bound. If either ablation matches the full method on the decisive checks below, that ingredient is not doing the work.

Why not just…?

Why not just clean the gender metadata better and rerun the threshold regression? If perfect labels existed, this would be correct and much of the measurement model would be unnecessary. But self-reported gender covers a small, non-random subset of authors. The S2AND results show that disambiguation errors concentrate on new and junior researchers, so cleaning removes error unevenly across exactly the period around a threshold crossing. Our audit sample directly measures how much residual error remains after the best available cleaning. If that error is negligible in every context, we will say so and drop the measurement dimension. That is a real possible outcome, and we would report it.

Why not just correct misclassification with a standard errors-in-variables adjustment and fit a threshold model? This is the most serious threat, and we include it as the ablation with $\Gamma = 1$. Where name-error rates are well estimated and confounding is weak, it may recover thresholds as well as our method. It cannot, however, separate a real step from glass-cliff selection, the pattern documented in Board independence, women directors and firm performance in Malaysian small-cap firms. The semi-synthetic glass-cliff test below is designed to show where it fails. If it does not fail there, the confounding bound is unnecessary for this question.

Why not just adjust for co-authorship network embeddings as a proxy for hidden confounders? Homophily-based proxies are attractive in scholarly data. But the method's guarantees assume a correctly observed network and no spillover between co-authors, and both assumptions fail in authorship graphs. Even in its own clean semi-synthetic test, the reported estimate stayed about twice the truth. We run it as a baseline and also feed its embedding as an observed covariate into our bounds, which tests whether it raises $\Gamma^*$.

Why not just use instrumental variables, such as staggered diversity mandates, as a source of exogenous variation? Where a credible mandate exists, such as Malaysia's 30% public-sector target (Factors affecting gender diversity in Malaysia), instrumental-variable estimates are worth reporting alongside ours. But a mandate that targets 30% shifts units to that one value. It therefore cannot reveal the shape of the curve elsewhere. The learned-instrument methods in the file need sample sizes most institutional panels lack.

Proposed Experiments

The decisive experiments are semi-synthetic. Real OpenAlex units keep their covariates, network, and integrity flags. Outcomes are simulated from known curves, and corruption is injected at audit-estimated rates.

Test 1: a real step blurred by noise. We plant a true step at 25% and add region-specific name misclassification. The fixed-effects threshold regression should recover a shifted or smoothed threshold. Our procedure should cover 25% and report a large $\Gamma^*$.

Test 2: a spurious step from selection. We plant no threshold but let units with poor pre-period outcomes acquire more women. Every baseline should report a step. Our procedure should report a $\Gamma^*$ near the calibrated strength of pre-period distress. The method counts as useful only if it passes both tests at once.

Test 3: thresholds that differ only through measurement. We plant identical thresholds across regions but give regions different misclassification rates. This checks whether naive analyses invent context-dependent thresholds and whether our correction removes them.

On real data, we report $\Gamma^*$ for the history-style topic-diversification threshold and for the conference cascade. We also report the fraction of between-context threshold variance that survives correction. A pre-registered outcome would change our view: if real-data $\Gamma^*$ values are large and context variation survives correction, critical mass gains credible causal support it currently lacks. If they collapse, the field's headline numbers should be reported as descriptive regularities only.

Gender is treated as binary here only because the source records encode it that way. The audit will record non-binary and unclassifiable cases, and the measurement model's drop rate absorbs them rather than assigning them.