Psychiatric neuroimaging tries to find connectivity patterns that track how ill a person is. Connectivity means the degree to which distant brain regions fluctuate together in a resting MRI scan. The target for these studies is almost always a single severity number from a clinician-administered scale such as the Hamilton Depression Rating Scale. That number is an opinion. Two trained clinicians interviewing the same patient often give different scores, and the field throws that disagreement away before any model sees the data. We propose to keep it and treat it as something to predict. Each patient's severity becomes a distribution over raters, not a point. The modeling question then splits into two parts. Does connectivity predict where the raters center, and does it predict how widely they spread?
The opportunity matters because the literature contains a puzzle that disagreement could explain. Some connectivity markers separate patients from controls very well but show no relation to symptom severity. Others correlate cleanly with severity. If ratings disagree most on exactly those patients whose brains sit between prototypical patterns, two consequences follow. Severity correlations are capped by rater noise in a way that is not uniform across patients, and observer disagreement partly reflects genuine biological ambiguity rather than carelessness. The core claim of this note is that this can be tested directly. A model that predicts a per-patient spread of rater scores from connectivity can tell us whether clinical disagreement has a neural correlate beyond what the symptom profile alone explains.
What the three directions contribute, and where the gap is
The functional-network literature supplies the brain features and the central tension. Family-based case-control study of homotopic connectivity in first-episode, drug-naive schizophrenia measured interhemispheric synchrony, meaning how closely mirrored regions in the left and right hemispheres move together. Reduced synchrony in sensory and motor regions classified patients with 92.86% accuracy but did not correlate with symptom scores on the standard schizophrenia rating scale (PANSS), which the authors read as a stable "trait" marker. Two other studies point the other way. Medial reward and lateral non-reward orbitofrontal cortex circuits change in opposite directions in depression found that medial orbitofrontal decoupling tracks Hamilton severity in 421 patients. Assessing Fine-Granularity Structural and Functional Connectivity in Children With ADHD found 78 connections tracking hyperactivity and impulsivity scores. All three treat the clinical score as clean ground truth. None of them can tell a marker that is truly unrelated to severity apart from one whose relation is hidden by rater noise. These papers motivate the proposal.
The diagnostic-granularity literature shows why moving from "ill or not" to graded severity is hard, and it supplies the ordinal tools. Deep Learning Based Prediction of Pulmonary Hypertension in Newborns reports held-out F1 falling from 0.78 for detection to 0.63 for three-level severity. Its labels came from a single cardiologist, so the drop cannot be split into task difficulty versus label noise. Tournament Based Ranking CNN for the Cataract grading shows that ordinal, ranking-style formulations beat plain regression on subjective grades: 68.36% versus 57.48% exact match. It also still trains on consensus grades. We reuse the ordinal idea and drop the consensus assumption.
The inter-observer literature supplies the decisive precedent, from outside neuroimaging. More is more: leveraging multi-rater information for whole slide images grading via virtual expert panel trained one model per pathologist on Barrett's esophagus biopsies, each case read by about 11 experts. Pooling these virtual raters raised average F1 against all annotations from 72.9% to 78.2%. It also cut expected calibration error from 0.183 to 0.050. Expected calibration error is the average gap between predicted confidence and observed accuracy. That paper shows disagreement carries usable signal. It also constrains our design in two ways. One model per rater needs hundreds of cases per rater, since its virtual raters only converged below about 200 annotations. And it discarded the "indefinite" cases, which are precisely the ambiguous ones we care about.
The research file contains no study that pairs functional connectivity with multiple independent severity ratings per patient. We found no reported result on Lacuna for predicting rater disagreement from brain data. The closest work is the virtual-panel study, which models rater disagreement in histology but has no brain-network component and never asks what makes a case contentious. The second closest is the biotype work in Neurobiology-based cognitive biotypes using multi-scale intrinsic connectivity networks in psychotic disorders. It clusters connectivity into subgroups that cut across diagnoses but treats cognitive scores as exact.
Core Claim
We claim that observer disagreement about psychiatric severity is partly predictable from resting-state connectivity, and that this prediction holds after controlling for the patient's mean score and symptom profile. If it holds, two things follow. Rater spread can serve as a training target that improves calibration of brain-based severity estimates. And "trait-only" markers should be re-examined, because their weak severity correlation may be concentrated in high-disagreement patients.
The model is a cumulative-link ordinal model, a standard way to predict ordered categories. It is extended so that the brain sets both the location and the width of the rater distribution:
$$P(y_{ir} \le k) = \sigma\!\left(\frac{\theta_k + b_r - s(x_i)}{\tau(x_i)}\right)$$
The symbols are defined as follows. Here $y_{ir}$ is rater $r$'s severity grade for patient $i$, and $k$ indexes the ordered grades. $\theta_k$ are learned cutpoints between grades, and $b_r$ is a learned bias for each rater, standing for that clinician's general leniency or harshness. The function $s(x_i)$ maps patient $i$'s connectivity features $x_i$ to a latent severity. The function $\tau(x_i) > 0$ maps the same features to a case-level ambiguity that widens or narrows the spread of rater grades. $\sigma$ is the logistic function.
Rater biases are shared parameters rather than separate networks, so the model stays trainable with a handful of raters and a few hundred patients. That is the regime where one-network-per-rater fails. The key ingredient is $\tau(x_i)$. Fixing $\tau$ to a constant gives an ordinal model with rater offsets and nothing more, and that is our main ablation.
Research Program
The first step is data, because the needed dataset does not exist in the file. We propose re-rating rather than re-scanning. In a depression cohort with resting-state fMRI and audio- or video-recorded structured Hamilton interviews, four to six clinicians independently re-score each recording. Each clinician is blind to the others and to the imaging. This yields a patient-by-rater score matrix at modest cost. The same design applies to ADHD symptom interviews. If any cohort we can access already holds parallel informant ratings, such as parent and clinician scores, it serves as a second test set. We do not assume that public benchmarks like ADHD-200 contain such ratings.
The second step is the features, and we deliberately start simple. We use region-averaged correlation matrices on a standard atlas. We add the two a priori marker families from the literature: homotopic synchrony, as in the schizophrenia study, and orbitofrontal-centred connectivity, as in the depression study. A learned spatio-temporal graph encoder is a later upgrade. An example is the adaptive graph network in Adaptive spatial-temporal neural network for ADHD identification using functional fMRI, which reached 80.5% binary accuracy across five ADHD-200 sites. We would only adopt such an encoder if it improves held-out likelihood.
We also guard against a specific artifact. Disambiguating brain functional connectivity showed that a change in correlation can arise purely from changes in signal amplitude or added noise. A sicker or more restless patient might therefore look "ambiguous" in the brain features simply through head motion or amplitude differences. So every model also receives regional signal variances, mean framewise motion, and site as covariates. We additionally report results for features residualised on these quantities.
The third step is analysis. We test whether $\tau(x_i)$ predicts observed rater spread on held-out patients. We test whether high-$\tau$ patients lie near boundaries between connectivity subgroups, using clustering in the spirit of the biotype work. And we test whether the severity correlation of trait-like markers becomes significant once we restrict to low-disagreement patients or weight patients by inverse spread.
Baselines
The naive reference point averages each patient's rater scores and regresses that mean on connectivity with ridge regression. This is what the depression and ADHD studies above effectively do, using a single rater. It sets the floor for both severity prediction and calibration. For the disagreement question, the naive reference predicts every patient's spread as the training-set average. Any claim that the brain predicts disagreement must beat this constant.
The strongest existing method for multi-rater grading is the virtual expert panel of More is more. It reports 78.2% average F1 and 0.050 expected calibration error on the Barrett's esophagus dataset (DEPP), against 0.183 for consensus training. We cannot transfer those numbers to brain data. Instead we re-implement the panel on our data: one small ordinal model per rater on the same connectivity features, averaged as in their full-panel strategy. It gets the same splits, the same feature set, and the same hyperparameter search budget as our model, with an equal number of tuning trials per method chosen on validation likelihood. We count our model as beating it if held-out negative log-likelihood of individual rater scores is lower and calibration error is no worse, with confidence intervals from patient-level bootstrap. A tie on likelihood with far fewer parameters would still be informative, but we would not call it a win.
The ordinal-ranking baseline is a cumulative-link model trained on consensus grades. It borrows the cataract paper's ordinal framing without its CNN-specific tournament machinery, and it isolates whether multi-rater targets add anything over ordinal structure alone.
The key ablation fixes $\tau$ to a constant while keeping rater biases. If this matches the full model on held-out likelihood and on prediction of spread, the central claim fails.
A further baseline predicts the spread from clinical information alone: the mean score, item-level profile, comorbidities, and medication. It decides whether the brain adds anything to disagreement prediction.
Why not just…?
Why not just average the raters and correct the brain–severity correlation for attenuation? The classical correction for measurement unreliability rescales a correlation using the estimated inter-rater reliability. It would answer "how much does rater noise cap the population correlation" with one number and no model. It is a real competitor for that narrower question, and we will report it alongside our model. But it assumes noise is the same for every patient. It cannot say which patients are contentious or whether contentiousness has a brain signature, and those are the claims we test. If the full model's per-patient spread predictions do no better than a constant on held-out data, then the attenuation correction is all that is needed. The ablation with $\tau$ fixed to a constant tests exactly that.
Why not just train one model per clinician, as the virtual expert panel does? That design worked on histology with hundreds of annotations per pathologist. Its authors note that virtual raters converged only below about 200 annotations, and they flag computational cost as panels grow. A re-rated neuroimaging cohort will have a few hundred patients and a handful of raters, so per-rater networks are likely to overfit. Shared rater offsets are the cheaper alternative. We do not assume this and run the panel as a baseline. If it wins on held-out likelihood, we adopt it and our contribution reduces to the disagreement analysis.
Why not just predict disagreement from the clinical scores themselves? This is the most serious threat. Rater spread on bounded scales depends mechanically on the mean: patients near the floor cannot be scored lower, so their spread is compressed. Spread may also simply reflect a mixed symptom profile. If the clinical-only baseline explains spread as well as connectivity does, the brain contributes nothing and the hypothesis is rejected. The decisive test is therefore incremental. Connectivity must improve prediction of spread over the clinical-only model on held-out patients, with mean score entered flexibly.
Why not just use a richer learned graph encoder from the start? Learned spatio-temporal graphs have helped binary ADHD classification. But with a small re-rated sample, a high-capacity encoder risks fitting site and motion structure. The amplitude-confound result above warns that such artifacts can masquerade as coupling. We begin with fixed features plus confound covariates and escalate only if validation likelihood improves.
Proposed Experiments
The primary experiment uses the re-rated depression cohort with patient-level cross-validation and a held-out site if the cohort spans more than one. The main metrics are held-out negative log-likelihood of individual rater grades, expected calibration error, and Spearman correlation between predicted ambiguity $\tau(x_i)$ and observed rater standard deviation. That correlation is reported as a partial correlation controlling for mean score, clinical profile, motion, and signal variance. The decisive check is that this partial correlation is reliably above zero. Its bootstrap interval must exclude zero, and it must exceed a null distribution built by permuting brain features across patients matched on mean score.
The second experiment returns to the trait-versus-state puzzle. For homotopic-synchrony and orbitofrontal features, we compare their correlation with consensus severity in low-spread versus high-spread patients, and we test whether inverse-spread weighting raises it. The prediction is directional. Markers reported as severity-unrelated should gain correlation in low-disagreement patients. If they do not, the trait interpretation from the schizophrenia study is strengthened, which is also a useful finding.
A methods validation runs on the DEPP histology data described in the virtual-panel paper, if access permits. There we check that our shared-offset ordinal model with learned ambiguity reaches calibration in the range that paper reports. This would be one point of contact with a published multi-rater result. Separately, we run a simulation that injects patient-dependent rater noise into real connectivity-severity data. The simulation confirms that the estimator recovers $\tau$ when the signal is present and returns null when it is absent.
The work fails cleanly in either of two cases: fixing $\tau$ to a constant costs nothing, or clinical features alone explain the spread. In either case, observer disagreement in psychiatric severity would be best treated as patient-independent noise, and that would itself justify the simpler attenuation correction.
References
- Family-based case-control study of homotopic connectivity in first-episode, drug-naive schizophrenia at rest
- Medial reward and lateral non-reward orbitofrontal cortex circuits change in opposite directions in depression
- Assessing Fine-Granularity Structural and Functional Connectivity in Children With Attention Deficit Hyperactivity Disorder
- Deep Learning Based Prediction of Pulmonary Hypertension in Newborns Using Echocardiograms
- Tournament Based Ranking CNN for the Cataract grading
- More is more: leveraging multi-rater information for whole slide images grading via virtual expert panel
- Neurobiology-based cognitive biotypes using multi-scale intrinsic connectivity networks in psychotic disorders
- Adaptive spatial-temporal neural network for ADHD identification using functional fMRI
- Disambiguating brain functional connectivity