Template Fits for TUNEL: Measuring Apoptotic Yield the Way Collider Physicists Measure Rare Decays

The TUNEL assay is the standard way to count dying cells in tissue. An enzyme attaches fluorescent nucleotides to every exposed end of broken DNA, and a person or script then calls each nucleus "positive" or "negative" with an intensity cutoff. The enzyme cannot tell apoptosis, the orderly self-destruction of a cell, from necrosis, autolysis, or DNA sheared while the tissue was being prepared. Every cutoff therefore mixes true signal with an unknown amount of look-alike background, and the reported "percent TUNEL-positive" comes with no error bar for that contamination. Experimental particle physics faced the same structure decades ago. At the BABAR experiment, a decay occurring once in $10^5$ events is not found by a single cut. Physicists model the full distribution of a discriminating score as a mixture of signal and background shapes, take those shapes from simulation and control samples, and fit the signal yield together with nuisance parameters that absorb the mismatch between simulation and data.

We propose to carry this method over directly. Each segmented nucleus is treated as an event. A classifier is trained only on standard TUNEL control preparations: chemically induced apoptosis, chemically or thermally induced necrosis, enzyme-free negatives, and nuclease-treated positives. The number of apoptotic cells in an experimental sample is then estimated by an extended unbinned likelihood fit of that classifier's score distribution, with template-shape nuisance parameters calibrated on uninjured "sideband" tissue. The output is an apoptotic yield with a confidence interval that already includes the necrotic-background systematic. The claim we test is narrow: on mixtures of known composition, this fit recovers the apoptotic fraction with lower bias and correctly calibrated intervals, compared with threshold counting and with simpler correct-the-classifier methods, especially when the necrotic contamination changes between training and use.

Core Claim

Throughout the TUNEL literature in the file, the measured quantity is a thresholded count. Kidney ischemia–reperfusion studies such as Ruimin Tian et al. and Xudong Xu et al. infer drug protection from fewer TUNEL-positive nuclei. The zebrafish drug-screening pipeline in Phenotypic analysis of images of zebrafish treated with Alzheimer's γ-secretase inhibitors quantifies death by merging channels, thresholding, and applying morphological rules. In injured tissue, necrotic DNA breakage is itself treatment-dependent. A drug that shifts death from necrosis toward apoptosis, or the reverse, can therefore move the thresholded count in either direction without changing true apoptosis. Our central claim is that apoptotic yield should be estimated as a mixture parameter, not counted. The method's value lies in three things a cutoff cannot provide: it handles more than two populations, it uses the whole shape of the score distribution, and it propagates template uncertainty into the reported interval.

The fitted quantity is a set of component yields $\nu_k$ for $k \in$ {apoptotic, necrotic/shear, unlabeled}. Each nucleus $i$ in a sample carries a classifier score vector $s_i$. Each component has a template density $f_k(s \mid \theta)$, taken from control preparations and deformed by nuisance parameters $\theta$. The extended likelihood is

$$\mathcal{L}(\nu,\theta) = e^{-\sum_k \nu_k}\prod_{i=1}^{N}\sum_k \nu_k f_k(s_i\mid\theta)\;\prod_j \mathcal{N}(\theta_j;0,1),$$

where $N$ is the number of segmented nuclei and the Gaussian terms constrain each nuisance parameter to its calibrated range. This is the standard collider template fit described in the BABAR overview. Profile-likelihood intervals on $\nu_{\text{apoptotic}}$ are the reported result.

Research Program

The program has four stages. Only the third is a methodological bet; the others are engineering needed to make the comparison clean.

Stage 1: Segmentation. Nuclei are segmented from the DAPI channel, the counterstain that marks all DNA. Crowded, clumped pyknotic (shrunken, condensed) nuclei are the known failure mode here. We use the local-constraint embedding method of Chen, Strauch and Merhof, which forces touching objects into orthogonal embedding directions and reported a Symmetric Best Dice of 0.8307 against 0.7972 for Mask R-CNN on CVPPP2017. Stardist is the alternative. Segmentation errors are treated as an additional template component: a "debris/merged" class built from the segmenter's low-confidence objects. They are not assumed away.

Stage 2: Templates from controls. A per-nucleus classifier is trained on crops of the TUNEL and DAPI channels, with labels coming only from which control well a nucleus came from. This mirrors training a BABAR boosted decision tree ensemble on simulated signal and background, as in the B → Kνν̄ search. The classifier's output distributions on held-out control nuclei become the templates $f_k$.

Stage 3: Sideband calibration. Controls in a dish do not look exactly like necrosis in an injured kidney, just as simulation does not match the detector. The 14% signal-efficiency systematic in the B → Kνν̄ search shows how large this mismatch can be even in a mature analysis. We therefore define tissue sidebands: regions or animals expected to have negligible apoptosis but realistic fixation and handling artifacts, such as contralateral uninjured kidney, sham-operated animals, or the untreated wells of a screening plate. The necrotic/shear template's shift and width nuisance parameters are fitted on these sidebands, and their uncertainties become the Gaussian constraints in the main fit. This step is what separates our method from simply rescaling a classifier.

Stage 4: Unsupervised adaptation (optional, explicitly tested). After the fit, per-nucleus sWeights are computed. An sWeight is a statistical weight, sometimes negative, whose average over many events reproduces the signal distribution. They are produced by the sPlot method that Borisyak and Kazeev (2019) analyze for machine learning. Those authors show that $\mathbb{E}[w\mid x] = P(\text{signal}\mid x)$ and that regressing sWeights with a sigmoid-bounded squared-error loss avoids the divergence that raw negative weights cause in cross-entropy. Borisyak and Kazeev (2020) confirm the same behavior for boosted trees on LHCb muon data. We would use this to retrain a TUNEL-channel model on unlabeled target tissue.

Both papers assume that the discriminating variable is independent of the training features within each class. Here the discriminating variable would be a DAPI-only morphology score, and the training features would be TUNEL-channel crops. Within apoptotic cells, chromatin condensation and labeling intensity both increase with stage, so this assumption may well fail. We therefore measure within-class correlation on the control wells before running Stage 4. If the correlation is substantial, Stage 4 is dropped and the proposal stands on Stages 1 to 3.

Baselines

Threshold counting (naive reference). Rolling-ball background subtraction, which removes slowly varying background by subtracting a smoothed version of the image, is followed by Otsu thresholding of mean TUNEL intensity per nucleus. This is what the zebrafish pipeline and the injury studies effectively report. It is run on identical segmentations so that only the quantification step differs. To be generous to it, the threshold is also tuned on the control wells to maximize apoptotic-versus-necrotic accuracy.

Classify-and-count. The Stage 2 classifier is applied directly, and nuclei with apoptotic probability above 0.5 are counted. This isolates whether the classifier alone does the work.

Adjusted classify-and-count (strongest competitor). This method takes the classifier's true- and false-positive rates estimated on controls and solves the resulting linear system for the class proportions. It is the simplest fix for the fact that a classifier's raw count is biased when class proportions differ from training. It uses exactly the same classifier, data, and controls as our fit, and differs only in that it uses a single operating point and no nuisance parameters. A tuned three-class version is included.

Density-map counting. SAM-Counter from CellFMCount adapts the Segment Anything image encoder with a density-regression head. It is trained to regress apoptotic-cell density using control-derived labels with the same training budget as our classifier. On its own benchmark it reports a total-cell-counting mean absolute error of 22.12, against 27.46 for CSRNet. That number concerns counting all cells, not apoptotic fractions; we found no reported result on Lacuna for apoptotic-fraction estimation under necrotic contamination, so no external number exists to beat.

Ablations. Three ablations isolate the key ingredient: the fit without sideband nuisance parameters, a binned fit using 10 bins of the same score, and a two-component fit that merges the necrotic and unlabeled classes.

Criterion for beating the baselines. We count the method as beating adjusted classify-and-count only if, on the shifted-contamination test sets defined below, its absolute bias in apoptotic fraction is at least halved and its 68% intervals cover the truth at a rate within 60–76%. Comparable bias with calibrated intervals would be a weaker but still reportable result.

Why not just…?

Why not just co-stain for cleaved caspase-3 or Annexin V? Cleaved caspase-3 is an apoptosis-specific enzyme; Annexin V binds a membrane lipid exposed early in apoptosis. Co-staining is the biologist's answer, and where antibodies penetrate well it is a strong alternative. We include caspase-3 gating as a comparator on the in vitro sets. It has two weaknesses for our purpose. Caspase activity is transient and precedes DNA fragmentation, so late apoptotic, caspase-negative nuclei get misassigned. In whole-mount zebrafish embryos and thick sections, antibody penetration is uneven. CellFMCount additionally documents that models fail to transfer across stains. We expect co-staining to win in monolayers and to lose in thick tissue, and the experiments are designed to show where that crossover lies rather than to claim universal superiority.

Why not just use rolling-ball subtraction and Otsu thresholds? These methods are fast and standard, and they serve as our naive reference. A single global cutoff cannot represent a necrotic population whose intensity overlaps the apoptotic one. The cutoff's error also changes with the necrotic fraction, which is the quantity that varies between treatment arms. The shifted-contamination sets measure this bias directly.

Why not just apply adjusted classify-and-count? This is the real threat, because it already corrects most of the prior-shift bias with one line of algebra. It differs from the template fit in two ways. It relies on fixed true- and false-positive rates measured on controls, and it reports no systematic uncertainty. If tissue necrosis looks like control necrosis, it should roughly match the template fit, and we would report that honestly. The decisive comparison is therefore on sets where the necrotic template is deliberately shifted, such as different fixation delays, where only the sideband-calibrated fit can adapt. If adjusted classify-and-count still matches under shift, our contribution shrinks to interval calibration.

Why not just train a better segmenter or counter end to end? Models such as Stardist, Mask R-CNN, or SAM-Counter learn whatever labels they are given. With control-derived labels they inherit control-versus-tissue mismatch and provide no handle on it. Better segmentation improves our Stage 1 but does not address mixture estimation, which is why density-map counting appears as a baseline on identical inputs.

Proposed Experiments

The primary testbed mimics how Borisyak and Kazeev built mixtures from the UCI Higgs data with a known virtual mass variable. Cultured cells are plated as separate wells: staurosporine-induced apoptosis, freeze–thaw and heat necrosis, nuclease-treated positives, enzyme-free negatives, and untreated cells. Each is imaged under TUNEL with DAPI and cleaved caspase-3 counterstaining. Nuclei pooled from held-out wells form in-silico mixtures with known apoptotic fractions from 1% to 40%. Two complementary validations follow. Physically mixed wells, where cells are combined at known ratios before fixation, test for interaction artifacts that in-silico pooling cannot capture. Shifted-contamination sets use necrosis induced by a different method, or fixed after a longer delay, than the training controls.

The metrics are bias and mean absolute error of the estimated apoptotic fraction, interval coverage, and the smallest detectable difference in apoptosis between two arms at fixed nuclei count, which measures statistical power. The decisive check is the shifted-contamination comparison against adjusted classify-and-count, together with the binned-versus-unbinned ablation, which shows whether keeping full score information matters at low apoptotic fractions as the BABAR overview argues it does for rare signals.

The secondary testbed is tissue. We reanalyze renal ischemia–reperfusion sections with sham-operated sidebands, following the designs of Tian et al. and Xu et al., and zebrafish embryos treated with a γ-secretase inhibitor, following the screen design of the zebrafish pipeline paper. Ground truth here is partial, so the checks are internal. Fitted necrotic yield should rise with fixation delay while apoptotic yield stays flat. Sideband-fitted nuisance parameters should transfer between animals in the same batch. Where caspase-3 does penetrate, it should agree with the fitted apoptotic yield. A tissue result that contradicts the in-vitro calibration, for example coverage collapsing in tissue, would indicate that the sideband assumption fails, and we would report it as the method's boundary.