Some neural networks solve their final layer exactly with linear algebra instead of learning it by gradient descent. The least-squares solution for that layer depends on the data only through two summary matrices, and both are sums over classes. We propose to keep those summaries separately for each class and recombine them with class-balanced weights. The result is an exact, gradient-free classification layer that treats a class of 77 Alzheimer's patients as seriously as a class of 206 patients with mild cognitive impairment. Because the solve costs only a small matrix inversion, the class weights and per-class offsets can be tuned exactly, rather than approximated by noisy reweighted mini-batch gradients. We pair this layer with an autoencoder encoder, a network that compresses each scan into a short code by learning to reconstruct it. The encoder is trained through the closed-form layer. Its code width becomes an explicit knob for a known tension: wider features approximate functions better but let majority-class noise swamp minority signal.
This matters because medical cohorts are small, skewed and multi-class. The two tools the field relies on both have gaps here. Closed-form "broad" classifiers already cut ADNI training time from about 2,800 seconds to about 1.5 seconds (Ma et al.), but they ignore imbalance. Imbalance-aware losses do help minority classes (Scholz et al.), but they need iterative training with high variance.
Core Claim
Write $h(x)\in\mathbb{R}^d$ for the encoder's code of input $x$, and $y_c$ for the target vector assigned to class $c$. For each class $c$, keep a running second-moment matrix $G_c = \mathbb{E}[h h^\top \mid c]$ and mean $m_c = \mathbb{E}[h \mid c]$. The class-balanced ridge head is then
$$W(\pi) = \Big(\sum_c \pi_c G_c + \lambda I\Big)^{-1} \sum_c \pi_c\, m_c\, y_c^\top,$$
where $\pi$ is a vector of class weights on the simplex and $\lambda$ is a ridge penalty. Setting $\pi_c = N_c/N$, the class frequency, recovers the ordinary pseudoinverse or ridge head. Setting $\pi_c = 1/C$, for $C$ classes, gives the exact minimizer of balanced squared risk.
Two further knobs stay closed-form. Scaling class-specific target vectors $y_c$ acts as a margin adjustment. Adding a constant per-class offset to the outputs acts as a logit adjustment. These mirror the two corrections that Mor & Carmon analyze for gradient-trained linear classifiers.
The claim has three parts. First, per-class statistics make the balanced solution exact, with no sampling or reweighting noise. Li et al. show that upweighting and upsampling are equivalent under full-batch gradient descent and diverge only through mini-batch variance. A closed-form solve over accumulated statistics is the full-batch limit, so that pathology disappears by construction.
Second, keeping $G_c$ as running averages across batches fixes the mini-batch instability that Galashov et al. counter with a proximal penalty. That instability is worst under imbalance, because a 32-scan batch from the glioma cohort may contain zero or one oligodendroglioma.
Third, because weighted squared loss is still quadratic in $W$, their envelope-theorem argument carries over. The encoder can be trained end-to-end without differentiating through the inverse.
The argument is exact only when $W$ is the optimum for the current batch. With running statistics it becomes approximate, and we will measure the gap. The payoff is that $\pi$ and the offsets can be chosen by directly maximizing validation balanced accuracy or Matthews correlation coefficient (MCC, a correlation between predicted and true labels that stays informative under skew). Each candidate costs one $d\times d$ solve. This is how we reconcile closed-form training with the non-quadratic imbalance-aware metrics that no closed-form method has handled.
Research Program
We build the system in three stages.
The first stage is a frozen-feature study. We fix an autoencoder code, compute the per-class statistics once, and trace the full balanced-accuracy and MCC surface over $\pi$, target scaling and offsets. We compare the chosen solutions against the ordinary pseudoinverse and against a logistic probe with logit adjustment. We also test a prediction that combines two results in the file. Needell et al. show random-feature error falls like $O(1/n)$ in width. Mor & Carmon show that growing dimension without compensation lets majority noise poison minority weights, pushing minority error toward 100% in their setting. Together they predict that, under the frequency-weighted head, minority sensitivity is non-monotone in code width $d$ while majority accuracy keeps improving. Under the balanced head, the curve should flatten or become monotone.
The second stage is end-to-end training. A 3D (for MRI) or 2D (for fundus) convolutional encoder is trained with two terms. One is the squared loss of the balanced closed-form head, with $W$ re-solved every step from running per-class statistics. The other is a reconstruction term with weight $\alpha$. The reconstruction term is also class-balanced. The file points out that unsupervised reconstruction on skewed cohorts spends its capacity on majority morphology, so we weight reconstruction error per class by $1/N_c$. The $\alpha$ term earns its place only if it helps in the small-data regime, where 77 AD scans are not enough to shape a supervised encoder alone. We test that directly.
The third stage is a hybrid for the known shortfall of squared loss relative to cross-entropy (Galashov et al. report it trails on ImageNet). We fit a temperature-scaled softmax on top of the solved outputs, which gives calibrated probabilities without breaking the closed-form solve.
Baselines
The naive reference is the unweighted closed-form head: frequency-weighted $\pi$ on the same encoder, which is exactly the broad-learning pipeline of Ma et al.. That paper reports 92.65% AD vs. NC, 93.58% AD vs. MCI and 84.68% MCI vs. NC accuracy, all binary. The file contains no reported three-class balanced accuracy for it on ADNI, so we rerun it ourselves on identical splits. Beating it means higher three-class balanced accuracy and AD sensitivity at equal training time.
The strongest iterative competitor is cross-entropy plus soft-F1 or soft-MCC loss with oversampling, from Scholz et al.. That paper reports up to a 12.7% relative improvement in balanced accuracy on its glioma dataset and up to +51.3% in early-glaucoma F1. Both figures are relative, and no absolute reported result was found on Lacuna for either dataset. It is strongest because it is the only method in the file that directly optimizes an imbalance-aware metric on these medical cohorts. We run it with the same encoder architecture, the same splits, the same number of hyperparameter trials (for example 30 per method), and the same GPU budget. Beating it means matching or exceeding its balanced accuracy and minority F1 with lower variance across five seeds. Matching it at a fraction of the tuning cost would also count, and we would report that as a cost result rather than an accuracy result.
The strongest closed-form imbalance competitor is GE-IFRVFL-CIL (Kumar et al.). It is a random-feature network with graph embedding, fuzzy sample memberships and imbalance-ratio sample weights. The file reports that it has the highest average AUC on binary ADNI tasks, but gives no numeric value. We run it on our encoder codes, pairwise and one-vs-rest.
We also run a per-sample weighted ridge head in the same pipeline, the RVFL/ELM analogue with weights $1/N_c$ applied per sample. Algebraically it matches our balanced $\pi$, but it lacks running per-class statistics and exact $\pi$/offset search.
Three ablations isolate the ingredients. The first swaps running per-class statistics for per-batch solves. The second fixes $\pi$ uniform versus searching it. The third sets $\alpha=0$ and swaps the autoencoder for PCA codes of equal width.
Why not just…?
Why not just train a CNN with cross-entropy plus soft-F1, as Scholz et al. do? This is a real threat. On moderate-size cohorts it may simply win. The evidence against it is indirect. Scholz et al. note that soft metric losses alone sometimes caused whole classes to be ignored, which is why they combine them with cross-entropy. Li et al. show that scalar upweighting inflates mini-batch gradient variance. Our head-to-head comparison at equal tuning budget settles the question. If Scholz's method wins on balanced accuracy and our variance advantage is small, the proposal reduces to a speed result, and we will say so.
Why not just fit logistic regression with logit adjustment on frozen features? Mor & Carmon derive the optimal corrections for exactly this setup, and on frozen features it may tie our head. The first-stage study tests that directly. Our distinct value would then lie in end-to-end encoder training through an exact balanced head. A logistic probe cannot provide that without iterating inside every training step. If the tie extends to the end-to-end setting, the closed-form component is not justified.
Why not just add per-sample weights to an ordinary ridge head? In full batch this is the same balanced solution, which is why it is included as a baseline. It differs in two ways. It does not give a stable per-step solve when minority classes are absent from batches. It also does not expose cheap exact search over $\pi$ and offsets. The per-batch ablation shows whether that stability matters in practice.
Why not just use PCA or a supervised encoder instead of an autoencoder? The data processing inequality argument (study) warns that unsupervised bottlenecks can discard discriminative detail. The $\alpha=0$ and PCA ablations test whether the balanced reconstruction term adds anything. If it does not, we drop it and keep a supervised encoder.
Proposed Experiments
The four cohorts come from the file. The first is ADNI three-class, 77 AD / 206 MCI / 151 NC, in the Ma et al. setting. The second is the glioma MRI dataset (roughly 80/12/8), with TCGA held out as in Scholz et al. The third is glaucoma fundus images across three stages. The fourth is PhysioNet/CinC 2016 heart sounds. On heart sounds the reference is a 1D convolutional network (Ryu et al.) reporting an overall score of 79.5, with 70.8% sensitivity and 88.2% specificity. Our target there is higher sensitivity at no loss of specificity.
Metrics are balanced accuracy, macro-F1, MCC, per-class sensitivity, seed variance and wall-clock training time.
We run three decisive checks. The first is the predicted width non-monotonicity: minority sensitivity against $d$ under frequency versus balanced $\pi$. The second compares minority F1 variance across seeds under running versus per-batch statistics. The third is the head-to-head against Scholz et al. at equal budget, which decides whether this is an accuracy contribution or only an efficiency contribution.
References
- Development and validation of a deep-broad ensemble model for early detection of Alzheimer's disease
- Imbalance-aware loss functions improve medical image classification
- An Analytical Model for Overparameterized Learning Under Class Imbalance
- Upsample or Upweight? Balanced Training on Heavily Imbalanced Datasets
- Closed-form Last Layer Optimization
- Random vector functional link networks for function approximation on manifolds
- Graph Embedded Intuitionistic Fuzzy Random Vector Functional Link Neural Network for Class Imbalance Learning
- Does the Data Processing Inequality Reflect Practice? On the Utility of Low-Level Tasks
- Classification of Heart Sound Recordings Using Convolution Neural Network