Networks trained with biologically plausible learning rules end up with a recognizable structure, and recent work explains that structure as a tug-of-war between two forces. Weight decay pulls every weight toward zero. The learning signal, plus the randomness of stochastic gradients, pushes weights outward. Where these forces balance, the weights take on that structure. In feedback alignment (FA), a learning rule that sends errors backward through a fixed random matrix instead of the transpose of the forward weights, the forward weights only come to agree with that random matrix when weight decay suppresses everything else. Separately, synaptic updates take on a "Hebbian" pattern, meaning they line up with the product of a neuron's input and output activity, when weight decay is strong relative to gradient noise. Strong noise flips the updates to the opposite, anti-Hebbian pattern. Curriculum learning, which trains on easy examples before hard ones, changes exactly the quantity these theories care about: easy data early produces smaller and more consistent gradients. Nobody has asked what that does to the force balance.
This note proposes a controlled test of a sharp question. When an easy-to-hard curriculum helps a feedback-alignment network, does it help only by lowering gradient noise? If so, a batch-size or weight-decay schedule should do the same job. Or do easy examples supply a consistent error direction that the fixed random feedback matrix can lock onto, something no noise schedule can imitate? We build a curriculum whose pace is set by measured alignment, and we compare it against three matched controls. Each control moves the noise-to-contraction balance by the same amount through a different lever: the data, the batch size, or the weight decay. If the curriculum's benefit under FA survives these matched controls while its benefit under ordinary backpropagation does not, curricula play a mechanistic role in local learning rules that goes beyond noise reduction. If it does not survive, we will have shown that curriculum effects in this regime reduce to a noise schedule. That outcome is useful too, and it would let practitioners drop curricula entirely.
Core Claim
The force-balance picture comes from Koplow et al. and Koplow et al.. Near stationarity the net update vanishes, so the learning signal must cancel the weight-decay pull $\gamma W$, where $\gamma$ is the weight-decay coefficient and $W$ a layer's weights. These papers find that the sign of the resulting Hebbian alignment is governed by a phase boundary $\gamma \propto \sigma^2$, where $\sigma^2$ is the variance of injected weight noise. They also find that Hebbian alignment shows a transient "bump" during early feature learning. We summarize the operating point of a layer by the ratio
$$\rho_t = \frac{\sigma_t^2}{\gamma_t},$$
where $\sigma_t^2$ is the per-step variance of the minibatch learning signal at training step $t$ and $\gamma_t$ the weight decay in force at that step. The Koplow phase boundary corresponds to a constant critical value of $\rho$. An easy-to-hard curriculum lowers $\sigma_t^2$ early. The same is true of a larger batch, since minibatch variance scales roughly as $1/B$ for batch size $B$. A higher $\gamma_t$ early lowers $\rho_t$ in the same way.
Our claim is that, under FA, the curriculum does more than lower $\rho_t$. Song et al. show that, in two-layer networks, regularization produces alignment by letting a persistent feedback-driven signal dominate the arbitrary initial weight components. Easy, low-diversity data makes that persistent component larger relative to the sample-specific part. We therefore predict that a curriculum produces higher forward–feedback alignment and higher final accuracy than a uniform-data control with the same $\rho_t$ trajectory. We predict this gap under FA and direct feedback alignment (DFA, which projects the output error straight to every layer through fixed random matrices), and we predict no such gap under backpropagation, where there is no random feedback matrix to align to.
Relation to Prior Work
Two papers motivate the alignment side. Refinetti et al. show that DFA learning proceeds in two phases: first the forward weights rotate toward the feedback weights, then the network memorizes the data, often giving up alignment to do so. In their analysis the switch between phases is passive and driven by uniformly sampled data. We make the switch an experimental variable. Robertson et al. prove a conservation law that ties a neuron's incoming-weight evolution to the alignment of its outgoing weights. They also report a large multi-class gap on 50 classes of CIFAR-100: 32.6% for standard FA, 48.9% for sign-FA, and 50.6% for gradient descent. Their law explains how alignment is kept once present but not how it first grows, which is exactly the regime a curriculum would act on.
Two papers constrain the design. Refinetti et al. also show that DFA fails in convolutional layers because weight sharing removes the freedom needed to align. We use this as a negative control: an easy-to-hard curriculum should not rescue DFA in convolutional layers. If it does, our mechanism is wrong. Koplow et al. report persistent anti-Hebbian layers in deep ResNets even under high weight decay. That is an anomaly our measurements can probe but that our theory does not presume to explain.
The remaining work supplies methods and comparison points. Understanding and Scheduling Weight Decay schedules weight decay from internal gradient statistics. We reuse its idea to build the weight-decay-matched control, but our schedule tracks a target $\rho_t$ rather than gradient norms alone. Self-Paced Learning for Latent Variable Models supplies a student-driven difficulty ranking that needs no external teacher. That matters here, because ranking examples with a backpropagation-trained model would contaminate a study of local learning rules. A Task-centric Theory for Iterative Self-Improvement with Easy-to-Hard Curricula gives three conditions for curricula to beat fixed mixtures: moderate separation between stages, a critical per-stage sample budget, and a feasible first stage. We use these conditions to choose stage count and budget, and we test the third directly. That paper's setting is reward-filtered fine-tuning of language models with no weight-space analysis. Ours is the reverse.
Enabling Fine-Tuning of Direct Feedback Alignment via Feedback-Weight Matching rescues DFA by building the feedback matrices from a singular value decomposition of pretrained weights. That fixes the feedback to match the forward weights. We instead change the data so the forward weights can find the feedback. The two are complementary, but we do not study fine-tuning.
Research Program
The method we build is an alignment-paced curriculum. Training data is split into difficulty tiers by self-paced loss under the FA student itself. We also run a class-incremental variant in which classes are introduced in groups. During the easy tiers, weight decay is held at an elevated value. We track each layer's cosine between forward weights and feedback weights. A harder tier is admitted only when this cosine has stopped rising, with "stopped rising" defined as a relative change below a fixed threshold over a fixed window. Weight decay then relaxes toward its base value. The pacing rule thus targets the Refinetti alignment phase and hands over to memorization only once alignment has formed.
Throughout training we log $\sigma_t^2$ per layer, using a held-aside fixed probe batch together with the per-sample gradient spread. We also log Koplow's Hebbian cosine per layer and the forward–feedback alignment cosine. From the curriculum run's logged $\rho_t$ trajectory we construct three controls, each on uniformly sampled data. The batch-matched control chooses batch size per phase so its measured $\sigma_t^2$ tracks the curriculum's. The decay-matched control keeps batch size fixed and sets $\gamma_t$ so that $\rho_t$ tracks the curriculum's. The combined control matches both. All controls see the same number of samples and the same total gradient steps, or the same sample count when batch size varies. We report both.
The theoretical piece is modest and targeted. In the two-layer teacher–student regression setting of Song et al. and Koplow et al., we will derive how the persistent component of the FA learning signal scales with teacher complexity, measured as the effective rank of the teacher's output map, at a fixed $\rho$. That derivation states the prediction quantitatively before the experiments run.
Baselines
The naive reference is uniform-sampling FA or DFA with a single constant weight decay, tuned over the same grid and seed count as every other method. Without this we cannot tell whether any schedule helps at all. On the 50-class CIFAR-100 setup of Robertson et al., the reported standard-FA number is 32.6% accuracy, and our reproduction must land near it before other comparisons count.
The strongest existing method for closing the FA gap is sign-FA from Robertson et al., which reaches 48.9% on that setup against 50.6% for gradient descent. We run sign-FA with the identical architecture, data, epoch budget, and tuning grid. Our curriculum is also applied on top of sign-FA, since the question is whether it adds something rather than whether it replaces a better feedback rule. Beating it means a curriculum-plus-sign-FA result above sign-FA's mean by more than the seed standard deviation, with a corresponding increase in alignment cosine.
For weight-decay scheduling we run AdamS from Understanding and Scheduling Weight Decay. It reports 4.91% test error for ResNet18 on CIFAR-10 under backpropagation, against 5.08% for AdamW. We use it as the strongest data-free schedule, with matched epochs. For curricula we run self-paced learning with its own annealed threshold, no weight-decay coupling, and no alignment gate.
The ablations isolate the key ingredient. The three $\rho$-matched controls above are the central ones. The curriculum without weight-decay coupling tests whether elevated early decay is needed. A random-class staging, with the same number of staged class groups but random rather than easy-first ordering, separates "fewer classes early" from "easier classes early." An anti-curriculum, hard tiers first, tests direction. A fixed-step pacing replaces the alignment gate to test whether gating matters.
Why not just…?
Why not just tune static weight decay, as with AdamW? Song et al. show that alignment depends on regularization, so a well-tuned constant decay is the obvious first fix. Understanding and Scheduling Weight Decay, however, reports that static decay tends toward sharp minima, with AdamW at 5.08% versus AdamS at 4.91% on CIFAR-10 ResNet18. A single constant also cannot be high during the alignment phase and low during memorization. Our naive baseline is exactly this, tuned on a full grid, so the comparison is direct.
Why not just schedule batch size or weight decay to lower noise early? This is the real threat. The Koplow phase boundary is stated purely in terms of noise and decay, so a noise schedule may reproduce everything a curriculum does. We do not argue around this; the $\rho$-matched controls are designed to settle it. If the batch-matched or decay-matched control equals the curriculum under FA, we will report that curricula act here only as a noise knob and recommend the cheaper schedule.
Why not just use sign-FA, adaptive initialization, or wider networks? Robertson et al. show sign-FA nearly closes the CIFAR-100 gap on its own, so the remaining headroom may be small. Width alone is not a fix: Song et al. show that unregularized alignment shrinks as $O(\sqrt{n/p})$ when width $p$ grows relative to sample count $n$. We treat sign-FA as a base rule to stack on rather than a competitor to beat. The scientific question is whether the curriculum's effect is separable from noise, and that question does not depend on the headroom.
Why not just use backpropagation with a learning-rate decay? If the goal were accuracy, backpropagation would win. The goal is to understand how training data shapes alignment under local rules. Learning-rate decay scales the decay pull and the gradient push together, so it does not move $\rho$ in the Koplow sense. We include a cosine learning-rate schedule in all runs so that it is held constant rather than confounded.
Proposed Experiments
The first stage is synthetic, using the teacher–student regression of Koplow et al. with 32-dimensional Gaussian inputs. Curriculum difficulty is set by teacher output rank or input noise, which lets us locate the $\gamma \propto \sigma^2$ boundary and test whether the curriculum shifts alignment at matched $\rho$. The second stage uses MLPs on MNIST and CIFAR-10 under FA, DFA, sign-FA, and backpropagation. The third stage is the 50-class CIFAR-100 setup of Robertson et al., with CIFAR-style CNNs under FA and sign-FA, plus DFA convolutional layers as the negative control.
Metrics are final test accuracy, per-layer forward–feedback alignment cosine, per-layer Hebbian cosine, and the fraction of layers in the anti-Hebbian regime. We use at least five seeds per cell and report confidence intervals.
Four checks decide the outcome. The central one is whether the curriculum beats the combined $\rho$-matched control in accuracy and alignment under FA, with no such gap under backpropagation. Second, DFA convolutional layers should show no curriculum benefit, consistent with the weight-sharing constraint of Refinetti et al. Third, following the feasible-initialization condition of Liu et al., starting at a hard tier should erase the benefit or cause regression. Fourth, the alignment-gated pacing should outperform fixed-step pacing at equal compute. That last check is the only evidence that measured alignment is a useful signal for scheduling rather than a passive symptom.
References
- Emergence of Hebbian Dynamics in Regularized Non-Local Learners
- Ubiquity of Emergent Hebbian Dynamics in Regularized Learning
- Convergence and Alignment of Gradient Descent with Random Backpropagation Weights
- The dynamics of learning with feedback alignment
- Implicit Regularization in Feedback Alignment Learning Mechanisms for Neural Networks
- Understanding and Scheduling Weight Decay
- Self-Paced Learning for Latent Variable Models
- A Task-centric Theory for Iterative Self-Improvement with Easy-to-Hard Curricula
- Enabling Fine-Tuning of Direct Feedback Alignment via Feedback-Weight Matching