When a neural network sorts detector events into "signal" and "background" at a false-positive rate near one in a million, the headline numbers it reports (signal events per year, signal-to-noise ratio) rest on two small numbers. The first is how many labeled background examples actually survive the cut. The second is how much the cut itself would move if the network's weights were slightly different but fit the training data equally well. Physics analyses handle the first with Poisson counting statistics. The second is usually ignored because the classifier is a single deterministic network. We propose to measure both cheaply and together. A post-hoc Laplace approximation turns the trained graph network into a distribution over plausible classifiers, and we combine that distribution with Poisson counting to put honest error bars on the sensitivity figure. A Laplace approximation fits a Gaussian over weights around the trained solution, using the curvature of the loss as the inverse variance. We then use the same posterior to decide which expensive labels to acquire next, choosing them to shrink those error bars fastest. The labels may be simulated events or annotated nodes.
The motivating case is the IceCube graph neural network of Graph Neural Networks for IceCube Signal Classification. It treats each event as a graph over active optical sensors and reports 5.772 signal events/year at a signal-to-noise ratio of 2.980, against 0.922 events/year for the physics baseline. Its background sample is about 366,000 unweighted simulated air showers. At a false-positive rate of order $10^{-6}$, the expected number of surviving unweighted background events is well below one. The research file does not say how the final-cut background rate was estimated, so we cannot judge that paper's numbers. The general point holds regardless: at these operating points, labels for the rare region are the scarce resource, and a deterministic network gives no sense of how fragile the cut is. The same structure appears in label-scarce node classification whenever one needs a small, high-purity set of positives from a large graph.
Core Claim
For a graph classifier run at extreme false-positive rates with scarce labels, the uncertainty in its sensitivity figure has two parts. One comes from weight uncertainty and is non-negligible. The other is Poisson counting. A last-layer Laplace posterior estimates the weight part with coverage comparable to deep ensembles, at negligible cost. Write $\varepsilon_b(\theta)$ for the background pass rate at a fixed score threshold under weights $\theta$, $\Phi_b$ for the physical background flux, and $\hat{b}$ for the estimated background count. Then the law of total variance gives
$$\mathrm{Var}[\hat{b}] = \mathbb{E}_{\theta}\big[\mathrm{Var}_{\text{count}}(\hat{b}\mid\theta)\big] + \Phi_b^2\,\mathrm{Var}_{\theta}\big[\varepsilon_b(\theta)\big],$$
where the outer expectation and variance are over the Laplace posterior $\theta \sim \mathcal{N}(\theta_{\text{MAP}}, H^{-1})$. Here $\theta_{\text{MAP}}$ is the trained weight vector and $H$ is the loss curvature there. The first term is the familiar Poisson or binomial counting error. The second is the classifier-fragility term that current analyses omit. The same decomposition applies to signal efficiency, and through both to signal-to-noise and discovery significance. Our second claim is about acquisition. Choosing new labels to minimize the expected posterior variance of this one scalar should beat accuracy-oriented acquisition, and it should beat the physicist's natural heuristic of "simulate more near the threshold."
Relation to Prior Work
The method comes from the post-hoc Laplace literature. Laplace Redux shows that last-layer Laplace recovers much of the calibration and out-of-distribution benefit of heavier Bayesian methods at almost no extra cost, and it provides the laplace toolkit we reuse. Scalable Marginal Likelihood Estimation shows the Laplace evidence can tune prior precision without a validation set. This matters when holding out labels would consume the few we have. That work also reports that the evidence can be brittle under misspecification, so we treat it as an ablation rather than an assumption.
The closest existing work is Laplace Sample Information. It scores each training sample by how much removing it shifts the Laplace weight posterior, measured as KL divergence, and finds that information is concentrated in a long tail of atypical samples. We borrow its efficiency move, curvature updates on a linear head over frozen features. We differ in the target. Laplace Sample Information measures information about all parameters under an i.i.d. assumption. We measure information about one downstream quantity: the background and signal pass rates at a fixed, extreme threshold. That quantity depends only on a thin slice of input space, so parameter-level informativeness is not the right score.
Three papers supply the problem rather than the method. The IceCube GNN and the EXO-200 analysis are deterministic selectors whose sensitivity claims are set by small counts. EXO-200 used a network on waveform images, reported a ~25% sensitivity gain from it, and validated against Monte Carlo simulation. Neither attaches weight uncertainty.
On the graph side, two lines of work constrain the design. FeatProp shows that model-based uncertainty fails in the first rounds of graph active learning. It avoids this by selecting nodes model-blind, from propagated features. Meta-GNN documents how badly standard GNNs fit at 1–5 labels per class. Both warn that a Laplace posterior around a poorly constrained solution may be miscalibrated, so we make coverage at small label counts an explicit test rather than an assumption. Sequential GCN active learning uses deterministic graph-classifier confidence for acquisition, which is the kind of score we expect to be overconfident early.
We deliberately avoid the hard problem of full-network curvature through message passing. In the IceCube setting each event is its own graph, so events are i.i.d. and the last layer is linear on a pooled embedding. In the node setting the last layer acts per node on frozen embeddings. Last-layer Laplace is therefore well-defined in both settings without new curvature backends.
Research Program
The first component is the estimator. We train the graph network as usual. We then fit a last-layer Laplace posterior, with prior precision set either by the Laplace evidence or by a fixed weight decay. For each posterior weight sample we compute the selector's pass counts at the operating threshold. We report a credible interval for events/year and signal-to-noise. This interval combines the posterior spread of efficiencies with the counting error on the labeled pool, using a Beta-binomial model rather than a Gaussian when surviving counts are near zero. For comparison we also fit a Kronecker-factored Laplace over the readout layers (A Scalable Laplace Approximation for Neural Networks). This checks whether the last layer misses fragility that lives in earlier layers.
The second component is sensitivity-targeted acquisition. A candidate's label enters the last-layer Gaussian as a rank-one update of the precision. We can therefore compute in closed form how much each candidate would reduce the variance of $\varepsilon_b$ and $\varepsilon_s$ at the threshold. In IceCube terms, a candidate is an event whose cheap features are known but whose expensive label or simulation is not. We emulate this by hiding most of the simulated pool and revealing labels on query. We greedily select batches and refit periodically.
The third component is a validity study of when curvature-based intervals can be trusted as labels grow. This addresses the cold-start concern head on. We test one specific remedy: initializing the label set with FeatProp-style representative nodes before switching to curvature-based acquisition.
Baselines
The naive reference point is the interval most analyses would report: a counting interval, Poisson or Clopper–Pearson, on held-out pass counts of a single trained network, with no weight uncertainty. If our intervals are no better calibrated than this, the proposal fails.
The strongest competitor for the weight-uncertainty term is a deep ensemble combined with bootstrap resampling of the training labels, scored by counting each member's pass counts. This is the expensive gold standard, and the honest question is whether Laplace matches its coverage. We run it with the same architecture, the same labeled pool, and the same hyperparameter search. We use five members, and report the ensemble's compute multiple against Laplace's. Laplace "beats" ensembles if its coverage of the true count is within a few points of the ensemble's at a fraction of the compute. It must not undercover at the smallest label budgets.
For predictive performance on IceCube, the reference is the reported GNN result of 5.772 signal events/year at signal-to-noise 2.980 (IceCube GNN). Our reproduction must land near this before any interval claims mean anything. For node classification the reported context is 65.27% 1-shot accuracy for Meta-SGC on Cora (Meta-GNN) and ~75% Macro-F1 for FeatProp with minimal labels (FeatProp). These are accuracy figures, not interval coverage. We found no reported result on Lacuna for interval coverage or interval width of a sensitivity figure on any of these benchmarks, so those comparisons are all run by us.
For acquisition, the baselines are random labeling, entropy or margin sampling of the deterministic network as in UncertainGCN (Sequential GCN), FeatProp selection, Laplace-Sample-Information-style parameter informativeness, and the threshold heuristic described below. All are given identical label budgets and refit schedules. Beating them means reaching a target interval width on events/year with materially fewer labels while keeping coverage. We set the threshold for "materially" in advance at 30%.
Ablations isolate the two ingredients. One removes the weight term, which leaves counting alone. One removes the counting term, which leaves the Laplace spread alone. One swaps evidence-tuned prior precision for fixed weight decay. One swaps last-layer curvature for Kronecker-factored readout curvature.
Why not just…?
Why not just report Poisson intervals and simulate more background? This is the real threat to the first claim. If the weight term in the decomposition is small relative to counting error, the Laplace machinery adds nothing. The variance decomposition is our first decisive experiment, at each label budget. If the weight term is under roughly a tenth of total variance across budgets, we will report that negative result and drop the estimation claim. More simulation still costs compute, so the acquisition question would remain.
Why not just use deep ensembles? They are the reference we must match, not a strawman. Laplace Redux reports Laplace as roughly competitive with ensembles on calibration at much lower cost. But that evidence comes from image benchmarks at moderate operating points, not $10^{-6}$ tails. Laplace's Gaussian around one solution may miss the mode-to-mode variation that ensembles capture, and that variation could dominate in the tails. The head-to-head coverage comparison settles this. If ensembles cover and Laplace undercovers, the conclusion is that tail sensitivity needs multimodal posteriors.
Why not just simulate or label more events near the score threshold? This heuristic is what a physicist would do, and it may capture most of the benefit, because the efficiencies only depend on inputs near the cut. Our acquisition score differs in one way. It weights candidates by how much they reduce disagreement among posterior classifiers about the cut. Score-near-threshold weights them only by proximity under one classifier. The two diverge when the threshold region is poorly constrained in some directions of embedding space. We test this directly as a baseline. If it ties our method, we will say so and recommend the heuristic plus Laplace intervals.
Why not just select labels model-blind with FeatProp? FeatProp is strong exactly in the cold-start regime where curvature is least trustworthy. Its objective, however, is representativeness for accuracy, not the rare tail that sets sensitivity. We expect it to win in the first rounds and lose afterward. The hybrid that seeds with FeatProp and then switches to curvature-based acquisition tests whether both can be had.
Proposed Experiments
The first experiment is a controlled node-level analog where ground truth is fully known. On Cora, Citeseer, PubMed, CoraFull, and Reddit, we designate one small class as "signal" and the rest as background. We train GNNs with 1, 3, 5, 10, and 20 labeled nodes per class. At each budget we fix a threshold giving a target false-positive count on the unlabeled graph. Because every label is known, we can check whether each method's 90% interval for retrieved signal and false positives contains the true values across many random seeds. The metrics are empirical coverage, interval width, and the variance decomposition.
The second experiment applies the same protocol to the IceCube Monte Carlo testbed. We first reproduce the reported GNN performance. We then subsample the ~8,500 signal and ~366,000 background events to emulate scarce simulation, and evaluate intervals on events/year and signal-to-noise against the full held-out simulation as truth. Background survival at the most extreme thresholds is too sparse to verify even with the full set. There we report coverage only at thresholds where the full set gives at least tens of surviving events, and we state explicitly that we are extrapolating beyond them.
The third experiment produces acquisition curves of interval width and coverage against label budget, on both testbeds, for all acquisition baselines.
Three checks decide the proposal. The first is whether the weight term is a material share of total variance. The second is whether last-layer Laplace intervals achieve coverage within a few points of ensemble-plus-bootstrap at 1–5 labels per class, or undercover as the cold-start literature predicts. The third is whether sensitivity-targeted acquisition beats the score-near-threshold heuristic by the pre-set margin. A negative on the first or third check is still informative. It would tell physics analyses either that classifier fragility can be safely ignored or that simple tail-focused simulation is enough.
References
- Graph Neural Networks for IceCube Signal Classification
- Laplace Redux - Effortless Bayesian Deep Learning
- Scalable Marginal Likelihood Estimation for Model Selection in Deep Learning
- Laplace Sample Information: Data Informativeness Through a Bayesian Lens
- Search for Neutrinoless Double-<mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML" display="inline"><mml:mrow><mml:mi>β</mml:mi></mml:mrow></mml:math>Decay with the Complete EXO-200 Dataset
- Active Learning for Graph Neural Networks via Node Feature Propagation.
- Meta-GNN: On Few-shot Node Classification in Graph Meta-learning
- Sequential Graph Convolutional Network for Active Learning
- A Scalable Laplace Approximation for Neural Networks