Scoring Neural Projectors Before Training With a Spectral Warm Start

Parametric dimensionality reduction trains a neural network to map high-dimensional data, such as images or single-cell gene profiles, to 2D coordinates. Unlike classical t-SNE or UMAP, the trained network can place new points with a single forward pass. The price is a large and poorly understood design space. Practitioners must choose depth, width, activation, normalization and loss weights, and today they do so by hand. Training is also non-convex, so the same architecture can give a clean or a scrambled map depending on the random seed. Searching this space by training every candidate is expensive: one parametric UMAP run on a 2-million-cell atlas did not finish in six hours (Navigating the Effect of Parametrization for Dimensionality Reduction).

This note proposes one cheap computation that serves two purposes. For each candidate architecture, take its untrained network and fit only the last linear layer, in closed form, so that it reproduces a spectral embedding computed on a small landmark subsample. Then measure how well the resulting map preserves neighborhoods on held-out points, and how much this varies across seeds. We claim this score ranks architectures by their final trained quality better than existing zero-cost scores and better than short training runs at the same compute. The fitted last layer is also a principled warm start for real training, and it reduces the seed-to-seed sensitivity that otherwise swamps differences between architectures.

Core Claim

Two observations from different literatures point the same way. Untrained networks already carry structure determined by their architecture (On Random Weights and Unsupervised Feature Learning shows convolution and pooling give frequency selectivity and invariance with random weights). In factored non-convex problems, a spectral initialization places the iterate inside a region where gradient descent converges well. Examples include the single power iteration of Fast Stochastic Algorithms for SVD and PCA, the SVD factor initialization in Finding Low-rank Solutions to Matrix Problems, Efficiently and Provably, and the spectral anchor in PhaseMax. A deep projector can be viewed as a fixed random feature map followed by a linear factor. If the architecture's random features can linearly reproduce a cheap nonlinear spectral target, the network starts near a good solution. If they cannot, no warm start helps it.

Concretely, let $H_{\theta_0} \in \mathbb{R}^{n \times p}$ be the penultimate-layer features of $n$ landmark points at random initialization $\theta_0$. Let $Z \in \mathbb{R}^{n \times k}$ be a $k$-dimensional Laplacian-eigenmap embedding of those landmarks. The readout and score are

$$W^\star = \arg\min_W \|H_{\theta_0} W - Z\|_F^2 + \rho \|W\|_F^2, \qquad S(a) = \mathbb{E}_{\theta_0}\big[\mathrm{kNN}(H^{\text{test}}_{\theta_0} W^\star)\big] - \beta\, \mathrm{sd}_{\theta_0}\big[\mathrm{kNN}(H^{\text{test}}_{\theta_0} W^\star)\big].$$

Here $a$ is the architecture, $\rho$ is a ridge constant, and kNN is 10-nearest-neighbor preservation measured on held-out points against input-space neighbors. The expectation and standard deviation run over a handful of seeds, and $\beta$ weights stability. Computing $S(a)$ takes a forward pass and a small linear solve, with no gradient training.

A second term addresses the tension between smoothness and cluster separation identified by Huang, Wang and Rudin. Smooth random networks may score well on neighbor preservation while blurring clusters. We therefore also report a separation statistic: the ratio of feature-space distances between mid-near pairs, sampled as in ParamRepulsor, and true neighbor pairs. We test whether including it improves ranking in the top tier.

Relation to Prior Work

The motivating work comes from parametric dimensionality reduction. Navigating the Effect of Parametrization for Dimensionality Reduction shows that network inductive bias causes cluster blurring and that SiLU beats ReLU. It repairs blurring with hard-negative mining, uses fixed hand-chosen MLPs, and names architecture design as open. ParaDime unifies parametric t-SNE, UMAP and MDS in a declarative grammar. It treats the architecture as user input, identifies manual tuning as a main burden, and uses PCA initialization as an empirical trick. Parametric t-SNE and Embedding with Autoencoder Regularization establish the loss families and the instability from random initialization. EAER adds a reconstruction term to prevent collapse. None of these papers ranks architectures or principles the initialization; we do both with one computation.

The search-cost literature supplies the scoring idea and constrains it. EZNAS evolves zero-cost scores for supervised image classification. It shows that hand-crafted scores transfer poorly and degrade in the top 10% of architectures, and that its evolved score largely behaves like a non-linear parameter counter. That last finding is a warning: in our setting, a score that rewards size or smoothness could pick blurry projectors. Our score differs because it is computed against a geometric target rather than class labels. Rethinking Performance Estimation in Neural Architecture Search formalizes short-budget training as a ranking estimator; we treat it as our main competitor, not as a component. NAS-Bench-201 documents both the collapse of weight-sharing supernets into skip connections and the strength of plain random search. It also gives us the template of an exhaustive lookup benchmark, which parametric dimensionality reduction lacks.

From the initialization literature we take the method, not a guarantee. The matrix results of Shamir and Park et al. are proved for linear factorizations. Provable accelerated gradient method for nonconvex low rank optimization shows the good region shrinks as conditioning worsens, which predicts our score will be weaker on data with small spectral gaps. We prove nothing for deep networks. Our readout is exactly a spectrally initialized last factor, and we test empirically whether the analogy holds.

The teacher-student distillation of Horan et al. trains a network to regress precomputed spectral coordinates for the whole dataset. We compute the spectral target only on a few thousand landmarks and use it to score and initialize. Full training stays end-to-end on the original manifold loss.

Research Program

The first deliverable is a lookup benchmark for parametric dimensionality reduction, built because none exists. The search space consists of MLP encoders, with depth from 2 to 6 and widths from 64 to 1024. Activations are ReLU, SiLU or GELU, and each network is a plain or residual stack with or without layer normalization, in roughly 500 configurations. Each configuration is trained to convergence with three seeds under two losses: ParamRepulsor's mid-near negative-sampling loss and parametric UMAP's cross-entropy. Training uses two initializations, standard random and our spectral readout warm start. The datasets are MNIST and COIL-100 from Huang et al.'s suite, Forest Covertype from ParaDime, and a 100k-cell subsample of the Cao et al. organogenesis atlas. The primary target is 10-NN preservation accuracy, following Huang et al. We add a cluster-separation measure, because neighbor preservation alone can hide blurring.

The second deliverable is the score itself, together with an analysis of what drives it. We vary landmark count, spectral dimension $k$ and ridge $\rho$, and test whether the score survives a change of manifold loss. The question is whether a score computed once predicts ranking under both ParamRepulsor and UMAP training.

The third deliverable is a study of initialization sensitivity. We measure the seed-to-seed Kendall's tau of trained architectures, which is the noise ceiling any score can reach, under random and warm-started training. If warm starting raises this ceiling, then initialization sensitivity is part of why architecture choice in this field has looked arbitrary. Once it is controlled, architecture choice becomes a better-posed problem.

Baselines

The naive reference is no search: ParamRepulsor's default SiLU MLP from Navigating the Effect of Parametrization for Dimensionality Reduction, trained with its own initialization. It is the strongest existing method on this exact question, having outperformed other parametric baselines on 10 of 14 datasets by 10-NN preservation. Per-dataset values are not itemized in the file, so we will reproduce them ourselves. Our selected architecture beats it only if its trained 10-NN preservation is higher on a majority of datasets by more than the seed standard deviation, without worse cluster separation.

The strongest search competitor is short-budget training in the spirit of Rethinking Performance Estimation in Neural Architecture Search. Each candidate is trained briefly with the real loss and ranked by its 10-NN preservation. For fairness, its wall-clock budget per candidate is set equal to the cost of our score, including the landmark eigen-solve, and then to two and four times that cost. Both methods tune their own few settings on MNIST only and are frozen for the other datasets. We beat it if our score's overall and top-10% Kendall's tau exceed its tau at equal cost on at least three of four datasets.

Hand-crafted zero-cost scores are the third baseline: Synflow, SNIP and NASWOT, with the dimensionality reduction loss substituted where a loss is needed. The fourth is EZNAS-A transferred unchanged. For reference, EZNAS reports Kendall's tau of 0.65 for EZNAS-A against 0.57 for Synflow and NASWOT on NAS-Bench-201 CIFAR-10. Those figures are for classification; there is no reported result found on Lacuna for any zero-cost score on parametric dimensionality reduction. Parameter count is included as a trivial score, since EZNAS-A behaves largely like one.

Full-training random search at a budget of 20 candidates gives the cost-quality frontier.

Ablations isolate the key ingredient. The first drops the readout and scores raw random features by kNN preservation, which tests whether the spectral fit matters. The second replaces the Laplacian-eigenmap target with PCA, which tests whether a nonlinear target matters. The third drops the stability term, and the fourth drops the separation statistic. For training, we compare warm-start initialization against ParaDime-style PCA initialization of the bottleneck and against plain random initialization.

Why not just…?

Why not just train each candidate briefly and rank by early quality? This is the real threat. Early training on these losses may already sort architectures well, and short runs are simple. Our matched-compute comparison settles it. If short training at equal cost matches our tau, the scoring half of the claim fails and we will report that. The warm-start half would remain testable through the seed-variance study.

Why not just use the default architecture? Huang et al. show that activation choice changes cluster separation substantially, so the space plausibly matters. But it may be that architecture spread is smaller than seed noise. The benchmark measures both directly. If the default sits within seed noise of the best architecture on most datasets, the honest conclusion is "don't search, warm start instead," and the proposal reports that.

Why not just apply Synflow or NASWOT? They measure gradient saliency or activation patterns tied to classification and say nothing about pairwise geometry. EZNAS also shows they degrade in the top tier even in their home domain. They are included as baselines, so this question is answered empirically rather than assumed.

Why not just distill from a full t-SNE or UMAP teacher? Running the teacher on all points reintroduces the non-parametric cost that parametric methods exist to avoid, and it is impractical at atlas scale. Our spectral target is computed on a few thousand landmarks. We use it only to score and initialize, never as the training objective.

Why not a weight-sharing supernet? NAS-Bench-201 documents collapse into skip connections under classification. Losses defined only by relative distances give even weaker anchoring. A supernet would add exactly the non-convex instability this note is trying to remove.

Decisive Checks

The central test is Kendall's tau, overall and within the top 10%, between the score and trained 10-NN preservation. It is reported for each dataset and each loss, against the seed-to-seed noise ceiling, and compared with short training at matched cost. The second test asks whether warm-started training lowers seed variance and raises the noise ceiling relative to random and PCA-bottleneck initialization. It also asks whether this gain shrinks on data with small spectral gaps, as the conditioning results of Li et al. predict. If it does not shrink, the analogy to the matrix theory is weaker than claimed. The third test transfers the top-scoring architecture from the 100k-cell subsample to the full 2-million-cell Cao et al. atlas. There we compare trained quality and wall-clock time against ParamRepulsor's default architecture.

The proposal fails if the score does not beat short training at equal cost and the warm start does not reduce seed variance. In that case we would conclude that untrained geometry is not informative for this task.