How Many Forking Paths Hide in a Sholl Curve?

Neuroscientists routinely claim that a drug "rescued branching" or that an injury made microglia "hypertrophic." The evidence is usually a Sholl curve. Sholl analysis draws concentric circles or spheres around a cell body and counts how many branches cross each one. Producing that curve requires a long chain of choices: how to threshold the image, which tracing software to use and with what settings, where to put the center, how far apart the rings are, which cells to drop, and which statistical test to run (one test per radius, a test on the area under the curve, cells as independent samples or nested within animals). These choices are almost never written down before the data are seen. Nobody has measured how much freedom they give an analyst to turn a null result into a significant one. This proposal measures that freedom on public neuron data. It then builds a small executable pre-registration, a frozen and hashed analysis specification that a deterministic program can re-run against deposited images, and tests whether it closes the gap.

The point is not to tell biologists to pre-register. The point is to find out which choices need locking. If most false positives come from the statistics, a fixed mixed-effects test is the cure and pre-registering image processing is wasted friction. If image processing and tracing choices matter just as much, locking the statistics alone fails, and an execution check becomes necessary. That decomposition is the result the field lacks, and it decides what a sensible standard should look like.

Core Claim

The claim has two parts, and both are testable. First, under a true null (two groups of cells drawn from the same population), an analyst who searches a realistic menu of Sholl processing and testing choices will reach $p<0.05$ far more often than 5% of the time. The inflation will split measurably between image-side choices (thresholding, tracing algorithm, centering, ring spacing) and statistics-side choices (per-radius testing, treating cells as independent, outlier rules). Second, an executable analysis card that freezes both sides, and is checked by re-execution rather than by reading, will bring the false-positive rate back near nominal. It will also catch deviations that text-only pre-registration misses.

To make "searching" precise, let $\mathcal{K}$ be the set of analysis paths an analyst might plausibly try, and let $p_k$ be the p-value from path $k$. The quantity of interest is

$$\mathrm{FPR}_{\mathcal{K}} = \Pr\big(\min_{k \in \mathcal{K}} p_k < \alpha \mid H_0\big),$$

which is the chance that at least one path "works" when there is no effect. We also report a more realistic sequential version, in which the analyst tries paths in a plausible order and stops at the first significant one. Comparing $\mathrm{FPR}$ for the full menu, the image-only sub-menu, and the statistics-only sub-menu gives the decomposition.

Relation to Prior Work

Three lines of work motivate this. On the morphology side, the dossier contains many studies that use Sholl curves or spine counts as primary endpoints: diabetic neuropathy (Tian et al.), hypothalamic gliosis in obese mice (Sa et al.), microglia after brain injury by sex (Datta Roy et al.), and growth-factor effects on calf sensory neurons (Fadda et al.). None of these studies freeze their processing choices in advance, and none were designed to quantify how sensitive their conclusions are to those choices. We use them only to define the realistic menu $\mathcal{K}$. We do not audit them.

The second line constrains the design: automated tracing is brittle. BigNeuron ran 35 tracing algorithms on about 30,000 image volumes and found that no single algorithm wins across image types. BigNeuron scored algorithms by geometric distance to expert skeletons. It did not ask whether tracing disagreements flip downstream biological conclusions, and that is the question we ask. WIS-NeuroMath automates concentric outgrowth counts and agrees with commercial tools. Its parameters are tuned by hand in a graphical interface, which is exactly the kind of unrecorded freedom we want to measure. On the descriptor side, persistence-diagram metrics prove that a persistence diagram (a summary that records where each branch begins and ends along the tree) contains the Sholl curve and more. They report that it raises nearest-neighbor classification on 379 Drosophila neurons from about 30.6% to about 58.7%. More information helps classification. It is an open question whether it reduces or enlarges the forking-path menu, since the descriptor function is itself a choice. We test this.

The third line supplies the method. Two prior systems freeze plans before evaluation. Segado et al. froze a video pipeline for cerebral palsy risk before opening a held-out "lock-box" test set (ROC-AUC 0.77 on lock-box versus 0.73 ± 0.05 in cross-validation). Zhang et al. used immutable JSON "experiment cards" with falsification clauses to keep an automated research loop honest. Two others ground verification in execution. MechEvalAgent re-ran interpretability repositories and showed that evaluators who could not execute code agreed with experts only about 45% of the time, against more than 80% with execution. It also found that agents may patch broken scripts unless locked out. Singh et al. argue that automated checkers should verify facts deterministically rather than judge. We take the card idea from the second pair of systems and the execution check from the first. We keep the verifier deterministic and non-agentic, so there is nothing for it to patch.

Our work differs from all of these in its target. None of them applies to cell morphology. None of them measures how much freedom exists before deciding what to lock, which is our first deliverable.

Research Program

The first component is a null-and-planted-effect testbed built from public data. Statistics-side forks are studied on skeletons. We take homogeneous cell types from NeuroMorpho.Org collections, including the Four Families and Human Brain Project sets used by the persistence paper. Cells are split at random into pseudo-"control" and pseudo-"treatment" groups. Where animal or specimen identifiers exist, we split by animal, so that pseudoreplication (treating many cells from one animal as independent samples) can be studied honestly. Image-side forks need raw images, so we use BigNeuron's Gold166 volumes with their expert skeletons. Each image is re-traced under a grid of thresholds and several BigNeuron tracers. For statistical power, we plant effects into skeletons: prune a fixed percentage of terminal branches, or shorten distal segments. This yields known effect sizes, and we measure how often each path detects them.

The second component is the menu $\mathcal{K}$. It is fixed in advance by reading the methods sections of the morphology papers above. Its axes are threshold method, tracing algorithm, soma-center rule, ring spacing, radial range, cell-exclusion rule, summary (per-radius, area under curve, maximum crossings, critical radius), and test (t-test or Mann–Whitney on cells, a mixed model with animal as random effect, or permutation). We pre-register this menu and the whole study on a public registry before running any null experiment. The study thereby obeys its own prescription.

The third component is the executable card. The card is a small machine-readable file. It records the container digest, tracer name, version and parameters, center rule, radii, exclusion rule, the single primary statistic and test, and a hash of the planned image manifest. A verifier is given the card, the deposited images and the reported number. It re-executes deterministically, compares results within a stated numeric tolerance, and flags any field where the reported analysis differs from the card. The verifier cannot edit inputs or code.

Baselines

The naive reference point is current common practice. This means one tracer with hand-chosen parameters, per-radius t-tests with cells as the unit, and no pre-specification. The analyst is modeled as a sequential search over $\mathcal{K}$. On a true null, its false-positive rate is the number every other condition must beat. No reported Sholl false-positive figure was found on Lacuna, so this number is itself a first result.

The strongest competitor on the false-positive question is a fixed statistical recipe with no pre-registration of image processing. The recipe is area under the Sholl curve, tested with a mixed model that nests cells in animals. Its free parameters (tracer and threshold) are still searched by the analyst. It runs on exactly the same splits and image grid as the full card. No reported result was found on Lacuna for this baseline on Sholl data. The full card beats it only if the full card's null false-positive rate is lower by more than Monte Carlo error at matched power on the planted effects.

For the verification question, the strongest existing reference is execution-grounded auditing in the style of MechEvalAgent. Its reported figures are more than 80% agreement with human experts given execution, and roughly 45% when blinded, from The Story is Not the Science. Those figures come from interpretability repositories, not morphology, so we treat them as a target rather than a directly comparable number. Our deviation-detection rate must exceed text-only review by a margin at least comparable to that gap.

For morphology agreement, we report tracer-to-gold fidelity alongside conclusion stability. BigNeuron's quality-prediction model reached $R^2$ of 0.661, and WIS-NeuroMath reported $R^2 \ge 0.80$ against commercial tools. Reporting both shows whether geometric agreement at those levels still allows conclusions to flip.

Two ablations isolate the key ingredients. The first compares a card that locks only statistics against one that locks only image processing, which gives the decomposition directly. The second compares a card checked by reading against a card checked by execution. This isolates whether execution, rather than freezing alone, carries the deviation-catching benefit.

Why not just…?

Why not just use a reporting checklist? Checklists are filled in after the results are known, so they cannot shrink $\mathcal{K}$ during analysis. The NeurIPS 2019 program (Pineau et al.) raised code availability to 74.4%, yet 36% of authors still called error bars "not applicable" while 87% claimed clear metrics. We test this directly. In the deviation study, reviewers given a checklist and the paper are compared against the verifier. The question is whether they catch seeded switches of tracer, threshold or test.

Why not just fix the statistics and skip pre-registration? This is the real threat. If pseudoreplication and per-radius testing cause most of the inflation, a mandated mixed-model test on area under the curve captures nearly all of the benefit at a fraction of the friction. The decomposition ablation settles it. If the statistics-only card brings null false positives near 5% across the tracer and threshold grid, we will report that pre-registering image processing is unnecessary for Sholl endpoints, and recommend the simpler standard. Our prior that it will not suffice comes from BigNeuron's finding that tracer behavior depends on image quality more than on biology. That prior is untested for downstream conclusions, so this baseline is not argued away.

Why not just report a full multiverse instead of locking anything? Reporting all paths, as in a specification-curve analysis, exposes fragility honestly. It does not deliver a single confirmatory test with a known error rate, and readers still pick the paths they like. We include it as a reporting arm. We measure how often the multiverse's majority conclusion matches the planted ground truth, compared with the card's single pre-specified test. If the multiverse is as accurate and as calibrated, it is a legitimate alternative, and we will say so.

Why not just switch to persistence diagrams, since they carry more information? Higher classification accuracy (58.7% versus 30.6% on Drosophila) is a statement about power, not about false positives. Persistence analysis adds its own choices, including the descriptor function and how diagrams are turned into vectors. We run the same null and planted-effect protocol with persistence summaries as the endpoint. This shows whether they enlarge or shrink the forking-path menu at matched power.

Decisive Experiments

The first decisive check is the null inflation table. It reports $\mathrm{FPR}_{\mathcal{K}}$ for the full menu, the image-only menu, the statistics-only menu, and each card condition, estimated over thousands of random splits with confidence intervals. The core claim fails if the full-menu false-positive rate is within a few points of 5%. In that case Sholl analysis is not fragile, and the rest of the program is moot.

The second check is propagation. On Gold166, we measure how often swapping between BigNeuron tracers whose geometric error to gold is similar flips the sign or significance of a planted 10–20% pruning effect. This answers the question BigNeuron left open.

The third check is deviation detection. We construct a set of analysis reports, some faithful to their cards and some with seeded deviations: a changed threshold, a dropped cell, a switched test, or a different tracer version. These are shown to text-only reviewers and to the deterministic verifier. We report detection rate, false-alarm rate, and time per report.

Together, the three checks answer one practical question: to keep a Sholl claim honest, what must be frozen, and must it be re-executed?