Premature Convergence as False Exclusion in Profile-Likelihood Scans

Physicists rule out dark matter candidates with null results. A telescope sees no excess gamma rays from a dwarf galaxy, and every combination of particle mass and annihilation rate that would have produced a visible excess is declared excluded. To make that call honestly, the analysis must also let the unknown "nuisance" quantities vary and pick their most favourable values. These include how much dark matter the galaxy contains, the background level, and how cosmic rays diffuse through the Galaxy. The technical name for this is profiling. Profiling is an optimization problem hidden inside every exclusion plot. When the optimizer stops early in a poor local optimum, it underestimates how well a hypothesis could fit the data, and it wrongly excludes that hypothesis. So premature convergence, the familiar failure of evolutionary search in which the population collapses onto a local optimum, does not just waste compute in this setting. It publishes exclusion regions that are too aggressive.

We propose to treat exclusion-limit computation as a population search over the plane that physicists actually report. That plane is spanned by the parameters of interest, such as dark matter mass and annihilation cross-section. The search keeps an archive with several distinct nuisance solutions in each cell of the plane. It spends evaluations where the exclude/allow decision could still flip. It also reports how often the same decision was reached independently by separate search lineages. The deliverable is a scanner and a benchmark that measures scanner-induced false exclusion directly. That quantity is currently not measured in the studies we build on.

Core Claim

Write $\theta$ for the parameters of interest (mass $m_\chi$, cross-section $\langle\sigma v\rangle$), $\nu$ for the nuisance parameters, and $L$ for the likelihood. The profile test statistic is

$$ q(\theta) = -2\ln\frac{\max_{\nu} L(\theta,\nu)}{\max_{\theta',\nu} L(\theta',\nu)}, $$

and $\theta$ is excluded at 95% confidence when $q(\theta)$ exceeds a critical value $q_c$.

A scanner that returns a value $\hat L(\theta) < \max_\nu L(\theta,\nu)$ in the numerator reports $\hat q(\theta) > q(\theta)$. Whenever $q(\theta) < q_c < \hat q(\theta)$, the point is falsely excluded. Premature convergence in the nuisance directions therefore has a signed, physically meaningful cost: it biases limits toward being too strong.

Our claim is that this error is concentrated in two places. The first is cells near the exclusion boundary. The second is regions where the best nuisance configuration switches between separated modes. An example of the second is a region where a large J-factor and a low background trade off against a small J-factor and a high background. A search that keeps several nuisance modes per cell and focuses on boundary cells should cut false-exclusion area at a fixed evaluation budget, compared with both nested sampling and grid-plus-local-optimizer profiling.

Relation to Prior Work

Work that motivates the proposal. Global fits map exclusion regions with multimodal nested sampling. Bertone et al. used MultiNest over the constrained supersymmetric parameter space to exclude the "Focus Point" region with XENON100 data. Cuoco et al. profiled a 16-dimensional cosmic-ray propagation space with MultiNest and GALPROP to disfavour thermal wino dark matter near 2.8 TeV. Both treat the sampler as a black box. Neither quantifies how much of the excluded region depends on the sampler having found the true profile maximum. A nested sampler is built to estimate a posterior, not to maximize over nuisances at each point of a reporting plane.

Dwarf-galaxy joint likelihoods have few nuisances but large ones. Abe et al. and Maggio et al. combine about 354 hours of MAGIC data with J-factor uncertainties of a factor of 2 to 3. This is the regime where profiling is easy, and it serves as our sanity case.

Work that supplies the method. We reuse several operators from the metaheuristics literature. The heavy-tailed Cauchy mutation and the "top-$p\%$ leaders" idea come from Choi et al.. That paper showed Cauchy steps escape traps that Gaussian steps do not, and that a decreasing sigmoid schedule beats linear ones for differential evolution. The idea of treating diversity as an explicit objective rather than a side effect comes from Gupta et al..

Work that constrains the design. Tang et al. report that fitness-distance balancing loses its advantage around 100 dimensions because Euclidean distances concentrate. Shi et al. note that covariance-learning diversity operators become too expensive in high dimension. Both results argue against measuring diversity in the full parameter space. We therefore measure coverage only in the low-dimensional plane of interest, where distance is meaningful and cheap. Inside each cell we keep distinctness only coarsely.

This is the main departure from the swarm optimizers surveyed, including Zhang et al., Ma et al. and those above. Every one of them seeks a single global optimum. Exclusion requires the correct maximum at every point of a plane, which is a different target.

Research Program

The scanner. The scanner maintains an archive over a grid in $(\log m_\chi, \log\langle\sigma v\rangle)$. Each cell stores up to $K$ elite nuisance vectors, kept distinct by a separation rule in prior-normalized nuisance coordinates. Each elite is tagged with the lineage it descends from.

Offspring are produced in three ways. The first moves a parent to a neighbouring cell, so that nearby cells warm-start one another; nuisance optima usually vary smoothly across the plane. The second applies Cauchy mutation to the nuisances. The third crosses over elites from adjacent cells.

The evaluation budget is weighted toward two kinds of cells. The first kind are cells whose current $\hat q$ lies within a margin of $q_c$. The second kind are cells whose stored best is implausibly worse than their neighbours'. Under smoothness, a cell whose best likelihood sits far below its neighbours' is a signature of an unfinished optimization.

The scanner stops once two conditions hold together. No cell's exclusion decision has flipped for a set number of generations. Each boundary cell's decision has also been reached by at least $r$ independent lineages. This lineage-agreement count is also reported as a per-cell diagnostic of premature convergence, alongside the exclusion contour.

The analysis. We will measure false-exclusion and missed-exclusion area against a reference computed with a budget far larger than any compared method's. We will report how these errors vary with nuisance dimension, with the number of nuisance modes, and with evaluation budget. We will also test whether the "suspicious cell" signal predicts reference-measured error on held-out problems.

Baselines

The naive reference. This is a raster grid over the plane of interest with a single-start local optimizer (Nelder–Mead) profiling the nuisances in each cell from the prior centre. It represents what a quick analysis does, and it sets the floor.

The standard practice. This is MultiNest run as in Bertone et al. and Cuoco et al.. The profile is read off as the maximum likelihood among samples falling in each cell. It is the incumbent in the literature this proposal addresses. We tune live points and efficiency on the same development problems and give it the same number of likelihood evaluations. We found no reported result on Lacuna for the profile-likelihood accuracy or false-exclusion area of MultiNest on any benchmark. The only numbers available are physics outcomes, such as the 95% upper limit of $\langle\sigma v\rangle \sim 1.9\times10^{-24}$ cm³ s⁻¹ at 1.5 TeV in Abe et al.. Our simulated dwarf-galaxy testbed will reproduce the order of magnitude of that limit as a sanity check, not as a competitive target.

The strongest practical competitor. This is a grid with warm-started multi-start profiling. Each cell is optimized from its neighbour's optimum plus $S$ random restarts, with $S$ chosen so the total budget matches ours.

A strong generic metaheuristic. This is ACM differential evolution (Choi et al.), run independently per mass slice to find the upper limit at each mass. On the CEC2017 suite, the strongest reported optimizer in the file is MRIME-CD with an average rank of 1.51 (Shi et al.). We will include it as a per-slice optimizer where its covariance cost allows.

Ablations. Four ablations isolate the key ingredients. The first sets $K=1$, which removes multiple nuisance modes per cell. The second allocates the budget uniformly instead of focusing on boundary and suspicious cells. The third removes cross-cell crossover. The fourth replaces Cauchy steps with Gaussian ones.

What counts as winning. Beating the baselines means a statistically clear reduction in false-exclusion area at equal evaluations across seeds, without an increase in missed-exclusion area. The comparison is against the warm-started multi-start grid, which is the bar that matters. If $K=1$ performs as well as full $K$, the mode-multiplicity claim fails.

Why not just…?

Why not just use MultiNest, the established tool with statistical guarantees? Its guarantees concern the evidence and the posterior. They do not cover the maximum over nuisances at each point of a reporting plane. A cell can be excluded simply because few posterior samples landed near its profile maximum. Nothing in Bertone et al. or Cuoco et al. measures this effect. That is the gap, not evidence that MultiNest fails. The equal-budget comparison will show whether the effect is large or negligible, and either answer is worth reporting.

Why not just grid the two parameters of interest and run warm-started multi-start local optimization in each cell? This is the real threat. When likelihoods are cheap and the nuisance space is small, as in the dwarf-galaxy case, we expect it to tie or win, and we will say so. Our argument holds only in two regimes. The first is where nuisance optima switch modes across the plane: warm-start chains then follow the wrong mode, which is premature convergence along the scan. The second is where evaluations are expensive, as with GALPROP, so random restarts cannot be afforded in every cell. The mode-switching synthetic suite and the $K=1$ ablation will decide whether keeping several modes per cell is necessary or merely redundant with restarts.

Why not just run a better off-the-shelf metaheuristic such as MRIME-CD or ACM-DE per mass slice? These optimizers win CEC rankings for locating a single optimum (Shi et al., Choi et al.). An upper limit, however, needs the profile at many cross-section values per mass, not one optimum. Independent slices also discard the correlation between neighbouring cells. We will run them per slice at equal total budget. If they match the archive scanner, the contribution reduces to the benchmark and the false-exclusion diagnostic, and we will present it that way.

Proposed Experiments

Synthetic suite with exact ground truth. The first testbed has a two-dimensional parameter-of-interest plane with an analytic signal model. The nuisance landscapes are built from CEC2017 hybrid and composition functions in 4 to 30 dimensions. They are shifted so that the location of the best nuisance mode changes with the parameters of interest, creating mode switches at known curves. The reference profile comes from exhaustive multi-start optimization. The decisive check is whether false-exclusion area for the warm-started grid rises sharply near the mode-switch curves while ours does not.

Simulated dwarf-galaxy joint likelihood. The second testbed is a four-target Poisson likelihood using the exposures in the file: Draco 52.1 h, Coma Berenices 49.5 h, Segue 1 157.9 h and Ursa Major II 94.8 h. Each target has a log-normal J-factor nuisance of width factor 2 to 3 and a per-target background normalization, fit over the $b\bar b$ and $\tau^+\tau^-$ channels. Here we predict that all competent methods agree. Disagreement would itself indicate a bug or an unexpected problem.

Cosmic-ray antiproton fit. The third testbed is a 16-dimensional antiproton fit following the setup of Cuoco et al., run on AMS-02 data with GALPROP within a fixed evaluation budget. The physics question is whether the wino exclusion near 2.8 TeV and the narrow windows near the 9.4 TeV quintuplet mass keep their status under a scanner that certifies lineage agreement. We expect the wino exclusion to survive. If the narrow windows move, that would show that scanner choice matters for a published physics claim.

Metrics. Across all three testbeds we report false-exclusion and missed-exclusion area, the error of the 95% upper-limit curve relative to the reference, evaluations to reach a target error, and the calibration of the suspicious-cell and lineage-agreement diagnostics against measured error.