Models that predict whether a gene is switched on or off from the chemical marks on its packaging proteins (histones) are routinely judged on accuracy and never on whether their confidence can be trusted. This is not laziness. Nobody knows how uncertain the model should be about any given gene, so there is nothing to compare its confidence against. We propose to construct that missing answer key using the biology of silencing. The standard five-mark input omits histone acetylation, a chemical tag whose removal by histone deacetylases (HDACs) directly closes chromatin and shuts genes off. A gene silenced by deacetylation alone can therefore look indistinguishable from an active gene to a five-mark model. If we hold acetylation out of the input but use it to build a reference model, we can calculate, gene by gene, how much the five-mark model ought to doubt itself. That per-gene doubt becomes the ground truth against which uncertainty methods are scored.
Put in two sentences: we hide one mark that is known to decide silencing, and that turns "how uncertain should this prediction be?" from unknowable into something we can estimate. We then use that target to test whether common uncertainty methods, and the evaluation metrics that need no ground truth, actually find the genes the input cannot resolve.
Core Claim
Consider genes whose five-mark profiles look nearly identical but whose expression is decided by acetylation at the promoter and enhancers. For these genes, the correct five-mark posterior is a mixture over the acetylation states such profiles actually occur with. Let $x$ be the five-mark binned profile, $a$ the withheld acetylation profile (H3K27ac), and $r(y{=}1 \mid x, a)$ a reference classifier trained with all six marks. The target posterior for any five-mark model is then
$$ p^*(y{=}1 \mid x) = \mathbb{E}_{a \sim p(a \mid x)}\big[\, r(y{=}1 \mid x, a) \,\big], $$
and the target uncertainty is its binary entropy $u^*(x)$. This approximates the true Bayes posterior well when $r$ is close to deterministic, meaning acetylation plus the other marks nearly settles the label. That condition can be checked directly by measuring $r$'s own residual entropy, so the approximation error of the answer key is reported rather than assumed.
The claim we test is that $u^*$ is a better yardstick than the alternatives. The first alternative is the binary correctness labels used by ground-truth-free metrics. The second is the trivial ambiguity of genes whose expression sits near the median cutoff. We expect $u^*$ to change the conclusions about which uncertainty method works on chromatin. We also expect the change to concentrate where the biology predicts: among silent genes that lack the repressive methylation marks H3K27me3 and H3K9me3, which are the genes most plausibly silenced by deacetylation alone.
Prior Work and Its Role Here
The prediction setup comes from three papers that also define the gap. DeepChrome bins five histone marks around each transcription start site into a 5×100 matrix and classifies expression as high or low across 56 Roadmap Epigenomics (REMC) cell types, with a reported average AUC of 0.80. AttentiveChrome adds attention over bins and marks. It withheld H3K27ac from training and used it only to check that attention lands on acetylated regions. We reuse that same withholding, but as supervision for uncertainty rather than as an interpretability check. DeepChrome 2.0 showed that a 12-parameter linear model matches the 645k-parameter CNN at an average AUROC of 0.77. This constrains us: any uncertainty advantage for deep models has to be shown, not presumed. None of the three reports calibration or uncertainty of any kind. We found no reported result on Lacuna for calibration or uncertainty quality on REMC.
On the uncertainty side, Legitimate ground-truth-free metrics proves two results about uncertainty scores. UQ-AUC, which asks whether correct predictions get lower uncertainty than wrong ones, is maximized by any score monotone in the true error probability. UQ-C-index is maximized by scores ranked in line with the gap between the model's decision and the Bayes-optimal one. The paper reports 94–99% correlation with hidden truth on synthetic data. It is our main competitor as an evaluation method, because if its metrics already track $u^*$ on chromatin, our construction is unnecessary. Revisiting UQ evaluation in language models shows that a shared confounder, response length, can bias both the uncertainty score and the correctness metric and so invert AUROC rankings. We expect the genomic analogues to be read depth, gene length and distance from the median cutoff, and we test for them explicitly. Supervised UQ with multiple annotations supplies the method template: treat a measurable spread, in that case disagreement among radiologists, as a target for uncertainty. Our version replaces annotator spread with spread over a withheld causal variable. Benchmarking UQ for protein engineering supplies both the list of methods and a warning: no method dominated, and ensembles were accurate but poorly calibrated under shift. AvUC, a training loss that penalizes confident errors and hesitant correct answers, and the softmax-fallacy critique motivate testing uncertainty methods trained into the model alongside raw softmax confidence.
The biology fixes where $u^*$ should be large. HDACs have no DNA-binding domain of their own and are brought to specific genes by transcription factors. SNAI1 recruiting HDAC1 to SNAI2 and RAD21 recruiting the NuRD deacetylase complex to interferon genes are examples: each strips H3K27ac to silence its targets. SNAI1 silencing also involves gaining H3K27me3, which the five marks can see. Deacetylation-only silencing has no such visible counterpart. Diverse silent chromatin states shows that silent chromatin is heterogeneous, splitting into an H3K9me2/H2A.Z-marked state, an H3K27me3 state and an H3K9me3/HP1 state. That supports stratifying silent genes by which repressive marks they carry.
Research Program
We first restrict to the REMC cell types that have H3K27ac alongside the five core marks. We have not verified how many such types there are, and coverage must be confirmed before committing. In those cell types we train the six-mark reference $r$ using the same bins and the same median split as DeepChrome. We estimate $p(a \mid x)$ in three independent ways: nearest neighbours over held-out genes in five-mark space, a conditional density model, and the DeepChrome 2.0 GAN conditioned on the five marks. The spread of $u^*$ across these three estimators is reported as the uncertainty of the answer key itself.
We use two settings. In the semi-synthetic setting, labels are sampled from a fitted generative model in which acetylation alone decides silencing for a known subset of genes, so $p^*$ is exact by construction. This mirrors the Gaussian-mixture validation of the ground-truth-free metrics paper, and it tests whether our estimator recovers the planted truth. In the real setting, labels are actual REMC expression calls and $u^*$ is estimated as above.
In both settings we score every uncertainty method three ways: rank correlation of its uncertainty with $u^*$, squared distance between its probability and $p^*$, and UQ-AUC and UQ-C-index computed from ordinary correctness labels. Every comparison is repeated within bins of read depth, gene length and distance from the median, to expose confounding of the kind the language-model paper documented.
Baselines
The naive reference point is the raw softmax confidence of a DeepChrome CNN trained with plain cross-entropy. This is what current practice reports, and any method that fails to beat it on rank correlation with $u^*$ has no value here.
The strongest predictive baseline is the DeepChrome 2.0 12-parameter linear model, read as a logistic posterior, with reported AUROC 0.77 on REMC. Its probabilities come from a simple, well-specified likelihood, so it may well be the best-calibrated model. Beating it means higher Spearman correlation with $u^*$ at equal AUROC, with confidence intervals from bootstrapping over genes.
The strongest uncertainty baselines are three. First, a five-member deep ensemble of DeepChrome CNNs. Second, Monte Carlo dropout and a Gaussian process on the 5×100 features, following the protein benchmark's set of methods. Third, post-hoc temperature scaling of the CNN, which fits one number on a validation set to soften or sharpen confidence. All methods share the same gene splits and the same cell types. Each gets the same hyperparameter search budget, set at twenty configurations selected on validation negative log-likelihood. Ensembles are compared both at equal member count and at equal total training compute.
AvUC is included as the strongest method that builds calibration into training.
Two ablations isolate the key ingredient, the withheld acetylation. In the first, $u^*$ is recomputed with H3K27ac shuffled across genes within the same cell type. In the second, $u^*$ is replaced by the five-mark nearest-neighbour label entropy with no reference model. If method rankings under either ablation match those under the true $u^*$, acetylation is contributing nothing and the construction fails.
Why not just…?
Why not just use UQ-AUC and UQ-C-index, which need no ground truth? Those metrics are proven to be maximized by the right ordering of uncertainty. However, they are computed from noisy binary labels, and they are sensitive to confounders that affect both the uncertainty score and the label. This is a real threat to our proposal, and the experiment settles it directly. If method rankings by UQ-AUC on real labels agree with rankings by $u^*$ both overall and within confounder bins, we will report that the ground-truth-free metrics suffice for chromatin. That would be a useful negative result, and it would retire our construction.
Why not just call genes near the median expression cutoff ambiguous? That captures how the label is defined, not whether the input contains the needed information. It is included as a competing target. We test whether $u^*$ still predicts which genes models get wrong after partialling out distance from the median. If it does not, the deacetylation story adds nothing beyond the arbitrary cutoff, and we will say so.
Why not just add H3K27ac to the input? For prediction that is the right move wherever the mark has been measured, and the six-mark reference shows how much it helps. The proposal's goal is different: to evaluate uncertainty methods in the common case where a deciding variable is unmeasured, and the five-mark setting is a real instance of that case. Whether the method rankings found here carry over to other withheld marks is tested by repeating the construction with H3K27me3 withheld instead.
Why not just use deep ensembles, the usual default? The protein benchmark found ensembles accurate but poorly calibrated under shift, while DeepChrome 2.0 suggests the chromatin task is nearly linear. Both point toward simple probabilistic models. We do not assume either result transfers. The ensemble is a full baseline under matched compute.
Proposed Experiments and Decisive Checks
The first check is validity of the answer key. In the semi-synthetic setting, the estimated $u^*$ must reach rank correlation with the exact $p^*$ comparable to the 94–99% the ground-truth-free metrics paper achieved against its hidden truth. On real data, the six-mark reference must be close to deterministic, and the three conditional estimators must agree with one another. If either fails, $u^*$ is too noisy to serve as truth and the project stops at that finding.
The second check is the biological prediction. Silent genes carrying neither H3K27me3 nor H3K9me3 should show markedly higher $u^*$ than silent genes carrying either mark. Methods that score well should place their uncertainty on exactly that group.
The third check is the core result. We ask whether the ordering of methods by $u^*$ differs from the ordering by UQ-AUC on real labels, and whether any difference shrinks once the analysis is restricted to genes with matched read depth and length.
A secondary, exploratory test uses the HCT116 cells with DNA methyltransferase knocked out, if the core marks can be assembled for them. Loss of DNA methylation there converts H3K9me3 domains into neutral chromatin. This lets us ask whether models whose uncertainty tracks $u^*$ at steady state also flag the genes whose chromatin state changes after the perturbation.
References
- DeepChrome: deep-learning for predicting gene expression from histone modifications
- Attend and Predict: Understanding Gene Regulation by Selective Attention on Chromatin
- DeepChrome 2.0: Investigating and Improving Architectures, Visualizations, & Experiments
- Legitimate ground-truth-free metrics for deep uncertainty classification scoring
- Revisiting Uncertainty Quantification Evaluation in Language Models: Spurious Interactions with Response Length Bias Results
- Supervised Uncertainty Quantification for Segmentation with Multiple Annotations
- Benchmarking uncertainty quantification for protein engineering
- Improving model calibration with accuracy versus uncertainty optimization
- I Find Your Lack of Uncertainty in Computer Vision Disturbing
- SNAI1 recruits HDAC1 to suppress SNAI2 transcription during epithelial to mesenchymal transition
- RAD21 amplification epigenetically suppresses interferon signaling to promote immune evasion in ovarian cancer
- Diverse silent chromatin states modulate genome compartmentalization and loop extrusion barriers