Does One Gradient Step Learn Features From Text? Testing Spiked-Feature Theory on Research vs. Non-Research Documents

A two-layer neural network can enter one of two regimes. In the "lazy" regime its first-layer weights barely move from their random starting values, so it behaves like a fixed kernel method. In the "rich" regime it reshapes those weights to fit the structure of the task. A line of theory isolates the transition between the two in a minimal setting. The network takes one large gradient step on its first layer, freezes it, and then fits only the output layer by ridge regression. In this setting the single step adds a low-rank "spike" to the random weight matrix, and that spike lets the network beat every fixed kernel. Every such result, however, assumes continuous Gaussian or Gaussian-mixture inputs. Nobody knows whether the predicted spike, learning-rate thresholds, and gains survive on sparse, discrete, heavy-tailed data such as bag-of-words text.

We propose to test this on a task that is useful and has a measurable nonlinear component: separating real research papers from the non-research items that scholarly indexes ingest, such as journal mastheads, proceedings front matter, committee rosters, obituaries, tutorial abstracts, and position papers. Most of these items are easy, and a keyword model catches mastheads. A hard subset (tutorial abstracts, position papers, memorial essays) uses the same academic vocabulary as research. It differs mainly in what is absent, namely results, numbers, and experimental framing. The note makes three specific, falsifiable predictions about one-step learning on text. It also builds a benchmark to check them, and it reports honestly where a linear model is all one needs.

Core Claim

Our first claim is that on text the one-step spike is, to leading order, the contrast between the two class means in token space, with each document weighted by its residual at initialization. The update can be written as

$$W_1 = W_0 + \eta\, u v^\top, \qquad v \propto \tfrac{1}{n} X^\top r,$$

Here $W_0$ is the random first-layer matrix, $\eta$ is the first-step learning rate, $X$ is the $n \times d$ document-term matrix, $r$ is the vector of initial residuals (roughly the labels when the network starts near zero output), and $u$ is a width-dimensional vector set by the second-layer initialization and the activation's derivative. This has two consequences. The learned direction is something a Rocchio-style centroid classifier already computes. Any advantage of one-step learning over a linear model therefore cannot come from finding a better direction. It must come from the nonlinear features the network builds along that direction. The theory predicts these features appear in stages as $\eta$ grows.

Our second claim is that the learning-rate threshold for the gain moves with the spread of the data. Asymptotic Analysis of Two-Layer Neural Networks after One Gradient Step under Gaussian Mixtures Data with Structure proves that feature-learning strength scales as $\beta \propto \eta \|\Sigma\|$, where $\|\Sigma\|$ is the largest eigenvalue of the input covariance, for Gaussian mixtures. If this transfers to text, the critical learning rate $\eta^*$ at which test error first drops below the lazy baseline should scale as $1/\|\Sigma\|$. Here $\|\Sigma\|$ depends on choices that text practitioners make routinely: title-only versus abstract versus full text, raw counts versus length-normalized TF-IDF, and keeping versus removing stopwords. The prediction is a log-log slope of $-1$ for $\eta^*$ against $\|\Sigma\|$ across these representations. This can be checked without any new proof.

Our third claim is that the nonlinear gain concentrates on the hard subset. We expect the gain to be near zero on mastheads and front matter, which a linear model separates, and positive on tutorial abstracts, position papers, and memorials. For those classes, membership depends on how strongly a document projects onto the research direction, and that dependence is nonlinear. One example is that "academic but not empirical" sits at an intermediate projection value. If the gain is zero on the hard subset as well, the applied claim fails. We say so in advance.

Research Program

The theoretical starting points are the one-step analyses of High-dimensional Asymptotics of Feature Learning: How One Gradient Step Improves the Representation and Asymptotics of feature learning in two-layer networks after one gradient-step. The first proves that a large step ($\eta = \Theta(\sqrt{N})$ for width $N$) beats the fixed-kernel lower bound. The second proves that the resulting spiked model strictly beats linear and kernel estimators. Beyond these, A Theory of Non-Linear Feature Learning with One Gradient Step in Two-Layer Neural Networks shows a staircase: with $\eta = \Theta(n^\alpha)$, a second outlier singular value appears once $\alpha > 1/4$ and captures quadratic structure. Higher-order spikes follow, and the paper validates the effect on MNIST. Spectral Evolution and Invariance in Linear-width Neural Networks supplies the matching picture of outlier eigenvalues leaving the random bulk past a critical rate. All of these assume Gaussian or Gaussian-mixture covariates and single-index targets. We take their predictions as hypotheses to test on text, and we do not claim to extend their proofs.

The work has three stages. The first stage builds the benchmark. We sample titles and abstracts (and full text where available) from S2ORC: The Semantic Scholar Open Research Corpus. That pipeline parses 81.1M records with GROBID and heuristics but has no learned research/non-research filter. We use high-recall string patterns ("Editorial Board", "Front Matter", "In memoriam", "Tutorial Abstracts", "Position:") to oversample candidates, then hand-label a few thousand documents into research versus six non-research subclasses. The subclasses follow types visible in the Lacuna index, such as IEEE Transactions on Fuzzy Systems publication information, Front Matter, Reviewer Acknowledgements, Remembering Don Bryant (1950–2024), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 5: Tutorial Abstracts), and Position: Stop treating `AGI' as the north-star goal of AI research. Because the candidates were pulled in with string patterns, those same patterns would trivially predict the labels on the easy classes. We therefore also label a uniform random sample, so that recall estimates on it are not biased by how candidates were found. Train, validation, and test splits are made by publisher and venue, so that a masthead's journal name or editor list cannot carry over between them. As a second testbed with a published number, we use the ACL section-function corpus of Enhancing Identification of Structure Function of Academic Articles Using Contextual Information. It has about 24,000 section titles in six classes and gives us multi-class, title-only text.

The second stage is a controlled synthetic bridge. Before touching real documents, we generate bag-of-words data from a topic model with Zipfian word frequencies and planted class structure. This lets us vary sparsity, document length, and $\|\Sigma\|$ independently. On this data we check two things. The first is whether the top singular vector of the one-step gradient aligns with the class-mean contrast as the equation predicts. The second is whether the $\alpha > 1/4$ staircase threshold still holds once inputs are sparse and non-Gaussian. We expect heavy-tailed document lengths to blur it, because length multiplies each document's projection, so we also test length normalization as a fix.

The third stage applies the same measurements to the real benchmark. For each representation we sweep $\eta$ over a logarithmic grid spanning lazy to rich, using the theory's two-stage protocol. In that protocol the first-layer step and the ridge readout use either independent batches (as the proofs require) or the same batch (as practice does). We record test Macro-F1 per subclass, the number of outlier singular values of the updated weights, and the alignment of the top singular vector with the centroid contrast. We also record the richness measure $D_{LR}$ from Decoupling Dynamical Richness from Representation Learning: Towards Practical Measurement. That paper shows that a richer network is not automatically a better one: its example reaches 100% train accuracy and 10% test accuracy by latching onto spurious features. Text has an obvious candidate for such features, the proper nouns on mastheads and rosters. We therefore map the top singular vectors back to the vocabulary. We then measure the share of their weight that falls on person and venue names, and relate that share to the cross-venue accuracy drop as $\eta$ rises.

Baselines

The naive reference point is the string-pattern filter that we also use to oversample candidates, scored on the uniform random sample. A second naive point is the majority-class predictor. The pattern filter is what a corpus engineer would deploy first. If it reaches high recall on the hard subset, the classification problem is largely solved and only the theory question remains. Beating it means higher non-research recall at matched precision on the hard subset.

The strongest cheap competitor is length-normalized TF-IDF with logistic regression, tuned over regularization strength and n-gram range on the same validation split. It uses the same features as the network, is convex, and the equation above implies it already contains the one-step direction. Beating it requires a statistically significant Macro-F1 gain on the hard subset across at least five seeds and two representations. Gains on easy classes do not count.

The lazy baseline is a random-features model of the same width with a ridge readout. It is the one-step network with $\eta$ near zero, using identical data, width, and readout tuning, so the comparison isolates the effect of the step. We also run a fully trained two-layer network, trained for many epochs with Adam and the same width. This measures how much of multi-step training's benefit one step captures.

The strongest existing method is a fine-tuned scientific language model, the S2ORC-pretrained SciBERT described in S2ORC: The Semantic Scholar Open Research Corpus. We give it the same labeled training set and a fixed hyperparameter budget comparable to the one-step sweep, and we report both accuracy and wall-clock time. We do not expect to beat it on accuracy. The question is what fraction of its hard-subset gain over TF-IDF the one-step model recovers, and at what fraction of its cost. On the research/non-research task itself we found no reported result on Lacuna for any method. On the ACL section-function corpus, the strongest reported number is a Macro-F1 of 0.9454 for a title-based CNN with a bidirectional context window of three, from Enhancing Identification of Structure Function of Academic Articles Using Contextual Information. We use it as a reference ceiling for title-only models rather than as a target.

The key ablation is a plug-in centroid model. We take the random-features model and add a bank of neurons whose weights are the class-mean contrast vector, each with its own random scale and bias, and leave the rest random with no gradient step. If this matches the one-step network, the "feature learning" on text is centroid estimation passed through nonlinearities, and Claim 1 is confirmed in its strongest form. If the one-step network still wins, the higher-order spikes carry information the centroid does not. A second ablation compares independent batches with the same batch, which tells practitioners whether the theory's data-splitting assumption costs accuracy.

Why not just…?

Why not just use TF-IDF with logistic regression? This is the real threat, and we expect it to win or tie on mastheads, front matter, rosters, and calls for papers. The theory gives no reason to expect otherwise there. The proposal's applied claim rests entirely on the hard subset, where academic vocabulary overlaps with research. Weakly-Supervised Scientific Document Classification via Retrieval-Augmented Multi-Stage Training reports that simple keyword and embedding similarity fall clearly behind richer representations for scientific topic classification: its multi-stage pipeline reaches 81.9% Macro-F1 on MeSH against 78.1% for the best baseline. That is a different task, so it only motivates the experiment. If the per-subclass comparison shows no hard-subset gap, we report that the nonlinear advantage predicted for Gaussian data does not appear on this text task. The scientific contribution then reduces to the spectral and threshold tests, which still stand on their own.

Why not just write rules? Rules catch documents that announce themselves, but position papers, tutorial abstracts, and memorials have normal author lists, abstracts, and venue metadata. S2ORC's own pipeline is rule-based and offers no non-research classifier. The uniform random sample measures directly how much the rules miss.

Why not just fine-tune a scientific language model? For accuracy alone, one should. This proposal does not aim to replace it, and the comparison is reported as a cost-accuracy curve. The one-step model's value is that its learned features have a closed-form description: the spike direction, the number of outliers, and their vocabulary loadings. That lets us test predictions about representation choice and spurious-name reliance, which a fine-tuned transformer cannot expose.

Why not just add the centroid direction as a feature and skip the gradient step? That is precisely the plug-in ablation above, and our own Claim 1 suggests it may suffice. We treat that outcome as a result rather than a failure.

Decisive Checks

The proposal stands or falls on four measurements. First, on synthetic Zipfian data and then on real documents, the cosine similarity between the top singular vector of the one-step update and the residual-weighted class-mean contrast should exceed 0.9. A low value would refute Claim 1. Second, across at least six representations spanning roughly an order of magnitude in $\|\Sigma\|$, the fitted log-log slope of the critical learning rate against $\|\Sigma\|$ should be near $-1$. A flat slope means the data-spread substitution does not transfer to text. Third, the one-step network should beat both lazy random features and TF-IDF with logistic regression on hard-subset Macro-F1 at the best $\eta$, while the plug-in centroid model is used to attribute that gain. A null result here falsifies Claim 3. Fourth, as $\eta$ grows past its optimum, the share of the spike's weight on proper nouns should rise together with the cross-venue accuracy drop. That would extend the richness-without-generalization finding of Decoupling Dynamical Richness from Representation Learning: Towards Practical Measurement to text, and give a practical rule for where to stop increasing the learning rate. The fixed sweep, five seeds, venue-held-out splits, and the uniform random sample are fixed before any test-set numbers are seen.