Classical discriminant analysis finds the directions in feature space that best separate classes. For a two-class problem it can return only one such direction. The reason is geometric: two class means span a single line. Many clinical imaging tasks are binary (diseased versus healthy), and a single axis throws away the fact that "diseased" often covers several distinct presentations. Existing fixes split each class into sub-clusters or use local neighbourhoods. They assume all the data sits in one place, and they build their extra directions from individual-level statistics, which is exactly what differential privacy (a formal guarantee that the output barely changes when any one patient's record is added or removed) forbids you from exposing.
We propose to get the extra directions from the private ensemble itself. Split the private data into disjoint partitions, as in PATE (private aggregation of teacher ensembles). Let each "teacher" fit an ordinary one-dimensional Fisher discriminant on its own partition. Then privately release the average outer product of the teachers' directions. Each teacher contributes a single unit vector, so any one patient moves at most one term of that average by a bounded amount. Because different partitions contain different sub-populations, their directions point differently, and the averaged matrix has rank far above one. PATE treats teacher disagreement as noise: it abstains when teachers disagree and applies corrections when partitions are heterogeneous. We claim that for binary tasks, the spread of teacher decisions is the missing discriminant subspace. Its eigenvalue spectrum also gives a privacy-aware rule for choosing the subspace dimension.
Core Claim
Let $w_t \in \mathbb{R}^{D'}$ be the unit-norm, shrinkage-regularized Fisher direction of teacher $t$, for $t = 1,\dots,T$, computed on frozen pretrained image features that have been compressed to $D'$ dimensions using public data only. We release
$$\tilde M = \frac{1}{T}\sum_{t=1}^{T} w_t w_t^{\top} + E, \qquad E \text{ symmetric Gaussian with entry scale } \sigma = \frac{\sqrt{2}}{T}\cdot\frac{\sqrt{2\ln(1.25/\delta)}}{\epsilon}.$$
Here $\epsilon, \delta$ are the privacy parameters. The factor $\sqrt 2/T$ is the Frobenius-norm change when one teacher's rank-one term is replaced. The top eigenvectors of $\tilde M$ form the private discriminant subspace. We keep eigenvectors whose eigenvalues exceed the noise floor $\approx 2\sigma\sqrt{D'}$, which is the typical spectral norm of a $D'\times D'$ symmetric Gaussian matrix. This rule is our answer to the rank-selection problem raised by Few-shot learning for inference in medical imaging with subspace feature representations. A student classifier is then trained on public images projected into this subspace, using labels from confident Gaussian noisy-max voting. Both releases are accounted jointly with Rényi differential privacy.
The claim is falsifiable. If the directions beyond the first reflect real sub-population structure, they should align with the discriminant directions of the underlying subtypes and improve area under the ROC curve (AUC) over a one-dimensional projection. If they merely reflect finite-sample estimation noise, they should look like noise, and they should vanish under i.i.d. partitioning as partitions grow.
How This Relates to Prior Work
The motivation comes from Few-shot learning for inference in medical imaging with subspace feature representations. That paper projects 512-dimensional ResNet-18 features into subspaces across 14 medical datasets. It finds discriminant analysis beats SVD in 14 of 14 cases and beats the raw feature space in 13 of 14. It also flags the one-dimensional ceiling for binary tasks as a real limitation. It assumes centralized, non-private data; we remove that assumption.
The rank-breaking methods supply the target but not the mechanism. Multimodal Linear Discriminant Analysis via Structural Sparsity assigns every sample its own mean and fuses them with a sparsity penalty. Multimodal oriented discriminant analysis splits classes by normalized cuts and raised recognition from about 56% (LDA capped at 22 features) to about 63%. Localized Sliced Inverse Regression uses local means, and Distance Metric Learning Versus Fisher Discriminant Analysis uses pairwise constraints in closed form. All of them build extra rank out of per-sample or per-pair quantities, and all assume one pooled dataset. We build extra rank out of per-partition quantities, which is what makes sensitivity bounded.
The research file raises a concern that higher rank forces sensitivity to scale with the number of modes. That concern does not apply here, because each teacher still contributes one bounded object. The rank comes from diversity across teachers, not from finer-grained outputs per teacher.
The privacy machinery is reused from Semi-supervised Knowledge Transfer for Deep Learning from Private Training Data and Scalable Private Learning with PATE, which aggregate only discrete votes. Hot PATE aggregates output distributions for text generation, not geometric objects.
Preserving privacy with PATE for heterogeneous data shows that non-i.i.d. partitions collapse vote consensus: student accuracy on MNIST at 0% similarity falls to 10.54% under standard PATE, versus 70.95% with its correction method. That paper treats heterogeneity as damage. Our proposal predicts that the same heterogeneity helps the subspace, even as it hurts the votes. The two effects must be measured jointly.
Private subspace estimation (On Differentially Private Subspace Estimation in a Distribution-Free Setting) is unsupervised and targets variance, not class separation.
Research Program
Building the pipeline. We extract frozen ResNet-18 features, following the few-shot medical paper. We reduce them to $D'$ dimensions with PCA on public, unlabeled images; this costs no privacy and shrinks the $\sqrt{D'}$ noise factor. Teachers are fit on disjoint private partitions, each producing a shrinkage Fisher direction. We then release $\tilde M$ and apply the spectral cutoff. Separately, we release confident-GNMax labels on public queries. Finally, a student is trained on the public data projected into the kept subspace.
Analysis. The analysis separates two sources of teacher spread.
The first source is sampling noise. Under i.i.d. partitions of size $n$, the spread in Fisher estimates contributes an excess to $M$ that shrinks roughly as $1/n$, the usual variance of a plug-in estimator. It is shaped by the inverse within-class scatter, not by class structure.
The second source is heterogeneity. Under subtype-heterogeneous partitions, the spread should persist as $n$ grows.
We derive the expected noise contribution and test a debiased variant: subtract a public-data estimate of the noise term before the eigendecomposition. We also examine a variant where each teacher contributes a rank-$r$ projector instead of one vector. This increases per-teacher information at a sensitivity cost of $\sqrt{2r}/T$.
Tradeoff. Privacy noise on $\tilde M$ scales as $\sqrt{D'}/(T\epsilon)$. Partitions also get smaller as $T$ grows, which makes each teacher noisier. We will map where the useful regime lies and report it honestly, including the settings where it does not exist.
Proposed Experiments
Data and binary tasks. We use MedMNIST v2 (MedMNIST v2). We collapse multi-class sets into binary tasks: PathMNIST tissue types grouped into normal versus abnormal, DermaMNIST into benign versus malignant groups, and OCTMNIST into normal versus disease. This construction gives known sub-modes, namely the original classes, to check recovered directions against. We also include the native binary tasks in MedMNIST v2 and, where available, the BrainTumor and CovidCT sets from the few-shot paper.
Partitioning. Each training set is split into a private half and an unlabeled public half. Private partitions are either i.i.d. or subtype-skewed, with each teacher seeing mostly one abnormal subtype plus normals.
Metrics. We report test AUC and accuracy at $\epsilon \in \{1, 2, 4, 8\}$ with $\delta = 10^{-5}$. We also report principal angles between the private subspace and the non-private multi-class discriminant subspace of the original subtypes, and the number of dimensions kept by the spectral rule.
Baselines
Naive reference points. The do-nothing reference is standard confident-GNMax PATE with teachers voting and a logistic student on the full $D'$-dimensional public features, with no discriminant projection. Beside it we run its one-dimensional counterpart: the same student on a private average of the teacher directions. That average is rank one and uses identical sensitivity accounting. This second baseline is the ablation that isolates our key ingredient. Any gain over it is attributable to using the spread of teacher directions rather than their mean. We count the idea as working only if the rank-$>1$ subspace beats this ablation in AUC at matched $\epsilon$ and $T$ on the subtype-skewed partitions.
Strongest threat. The strongest competitor is post-processing on the public side. We run PATE labels, then the structural-sparsity multimodal LDA of Multimodal Linear Discriminant Analysis via Structural Sparsity on the noisily labeled public data. This is free under differential privacy's post-processing property and can also break the one-dimensional ceiling. It receives the same public data, the same labeling budget, and the same hyperparameter search size. We also test a combination, feeding our subspace into that method, since the two may be complementary.
Other competitors. A DP-SGD linear probe on the private features represents direct private training. We run it at equal $\epsilon$ with a matched tuning budget, though the file gives no reported result on Lacuna for DP-SGD on binary MedMNIST tasks. We also include a privatized-covariance closed-form metric learner, which adds Gaussian noise directly to the $D'\times D'$ scatter matrices and then solves the eigenproblem of Distance Metric Learning Versus Fisher Discriminant Analysis. Private PCA on the private features serves as the unsupervised rank-unconstrained reference.
Non-private ceilings and reported numbers. The ceilings are centralized binary discriminant analysis, which is one-dimensional, and centralized multimodal LDA. For orientation only, MedMNIST v2 reports an average 2D AUC of 0.927 for Google AutoML Vision. That figure is non-private and full-image. The few-shot paper reports about 58.7% accuracy for discriminant analysis on 9-class PathMNIST, against about 34% in the raw space. No reported result was found on Lacuna for any private method on binary MedMNIST tasks, so all private comparisons are run by us under identical conditions.
Why not just…?
Why not just let the student find sub-clusters on public data after PATE labeling? This is a real threat, and it may match us when public data is plentiful and its sub-modes are visually obvious. It is also free under post-processing. The difference is informational. Public-side clustering sees only binary labels and public feature geometry, so it finds modes of appearance. The teacher spread reflects how the label boundary itself rotates across private sub-populations. That information exists only in the private labels and is not recoverable from binary votes. The principal-angle test settles this. If public-side multimodal LDA aligns with the subtype discriminant subspace as well as ours at equal $\epsilon$, we will report that post-processing suffices.
Why not just use private or public PCA? PCA is rank-unconstrained but class-blind. The few-shot paper found SVD lost to discriminant analysis in all 14 datasets, and sometimes fell below the raw feature space. We use public PCA only as a free pre-compression step and include private PCA as a baseline.
Why not just run DP-SGD on the frozen features? A linear probe is cheap, and DP-SGD's per-step noise matters less for a convex model. So this baseline could be strong at large $\epsilon$. PATE's reported advantage (98.5% on MNIST at $\epsilon = 1.97$ in Scalable Private Learning with PATE) was measured for deep networks, not linear probes. We therefore treat this as an open empirical question and run it head to head.
Why not just relabel the task as multi-class so ordinary LDA gives more dimensions? The clinical labels are binary; subtype labels usually do not exist. Where they do, PATE voting over more classes splits consensus, which is the many-class problem that motivated confident aggregation. Our collapsed-MedMNIST tasks include this "oracle subtype" variant as an upper reference.
Why not just perturb the scatter matrices directly? Here each patient affects the scatter matrix itself, so noise must be calibrated per record in $D'\times D'$ entries. In our mechanism, each patient affects one of $T$ unit-norm terms. The privatized-covariance baseline measures this gap directly.
Decisive Checks
Three results would decide the proposal.
The first is subspace alignment. On subtype-skewed partitions, the private subspace should have small principal angles to the non-private subtype discriminant subspace. Under i.i.d. partitions, the extra eigenvalues should shrink toward the predicted sampling-noise level as partition size grows. If they do not shrink, the extra directions are artefacts.
The second is utility. The rank-$>1$ student should beat the rank-one ablation in AUC at matched $\epsilon$ and $T$, and should at least match public-side multimodal LDA.
The third is rank selection. The dimension chosen by the spectral noise-floor rule should fall within the range where held-out AUC plateaus.
If the first check fails, teacher spread is noise and the idea is wrong. If only the second fails against public-side clustering, the contribution narrows to a negative result about where private label geometry adds value.
References
- Few-shot learning for inference in medical imaging with subspace feature representations
- Multimodal Linear Discriminant Analysis via Structural Sparsity
- Multimodal oriented discriminant analysis
- Localized Sliced Inverse Regression
- Distance Metric Learning Versus Fisher Discriminant Analysis
- Semi-supervised Knowledge Transfer for Deep Learning from Private Training Data
- Scalable Private Learning with PATE
- Hot PATE: Private Aggregation of Distributions for Diverse Tasks
- Preserving privacy with PATE for heterogeneous data
- On Differentially Private Subspace Estimation in a Distribution-Free Setting
- MedMNIST v2 - A large-scale lightweight benchmark for 2D and 3D biomedical image classification