When a network is trained by stochastic gradient descent (SGD), the order in which examples arrive is usually treated as noise to be averaged away. Several observations do not fit that view. Reordering clean CIFAR-10 batches can drag a model from about 90% to 26% test accuracy. Data curricula, which present easy examples before hard ones, help BERT on GLUE but hurt T5 on translation. A 1991 pattern-selection rule traps small networks in endless weight-space orbits rather than in local minima. This note proposes a single mechanism behind these cases.
Every gradient step is a step downhill on one example's loss. When those steps are chained in a fixed order, the combined effect over an epoch is no longer a pure downhill move. To second order in the learning rate, the combined move contains a rotational part. Its size is set by how each example's gradient changes the next example's gradient, and its sign flips when the order is reversed. We call this term the order curl. The proposal is to measure it in real networks, test whether it predicts when structured orders cause oscillation or orbit trapping, and test a cheap fix. The fix is to run each epoch's order forward and then backward (a palindrome). This cancels the curl exactly at leading order and leaves the curriculum's choice of data untouched.
Core Claim
Write the SGD update as $\theta_{k+1} = \theta_k - \eta\, g_{\pi(k)}(\theta_k)$. Here $\eta$ is the learning rate, $g_i$ is the gradient of example (or batch) $i$'s loss, and $\pi$ is the order of the $n$ examples in an epoch. Let $H_i$ be the Hessian of example $i$'s loss. Expanding the epoch map around its starting point gives
$$\Delta\theta = -\eta \sum_i g_i + \frac{\eta^2}{2}\nabla\!\sum_{k<l} g_k\!\cdot g_l + \frac{\eta^2}{2}\sum_{k<l}\big(H_{\pi(l)}g_{\pi(k)} - H_{\pi(k)}g_{\pi(l)}\big) + O(\eta^3).$$
The first two terms do not depend on order. The second is the gradient of a potential, the familiar implicit regularizer that penalizes gradient alignment. The third term, which we denote $A_\pi$, is a sum of Lie brackets of the per-example gradient fields. A Lie bracket measures how much two flows fail to commute, so it is generally not the gradient of any function. It is the centralized-training analogue of the conservatism breakdown that produces limit cycles in federated averaging.
Three consequences follow, each of which can be tested:
- Reordering a fixed set of examples adds only rotation, at second order. Whatever a reordering changes at that order, it cannot add descent. Any benefit a curriculum gives must therefore come from pacing, meaning which examples appear and how often, or from higher-order effects.
- Reversal flips the curl, and a palindrome cancels it. Reversing an epoch flips the sign of $A_\pi$ while leaving every variance statistic of the batches unchanged. A palindromic epoch has $A=0$ exactly, because the within-half terms cancel and every cross pair appears in both orders.
- Random reshuffling has zero curl only on average. Under random reshuffling, $A_\pi$ is zero in expectation but has nonzero spread. Structured orders, whether curricula, loss-sorted oscillations or block cycles, give it a persistent sign. We predict that this persistent component drives the orbiting behaviour.
Relation to Prior Work
Two papers supply the dynamical lens.
- Iterated Vector Fields and Conservatism, with Applications to Federated Learning proves that composing local gradient steps breaks conservatism and can trap federated averaging in stable limit cycles. It studies deterministic full-batch client updates and never asks how ordering within a sequence shapes the rotation. That ordering question is the one we take up.
- Stochastic gradient descent performs variational inference, converges to limit cycles for deep networks attributes SGD's rotational dynamics to highly anisotropic noise, with gradient-covariance rank of 0.3% to 2% of parameter dimension. It works in a continuous-time limit with sampling with replacement, where order has no meaning. Our term is a discrete-step, order-specific source of rotation that exists even without noise.
Four papers motivate the proposal. Each observed order-dependent trajectories without a mechanism.
- Repeat Until Bored: A Pattern Selection Strategy repeats a pattern until its error drops below a threshold. On parity tasks it found "dead zones" near threshold 0.25 where weights enter non-optimal orbits, worse with momentum. It offers no theory and uses tiny networks only.
- Manipulating SGD with Data Ordering Attacks builds loss-sorted "oscillation" batch orders that cut CIFAR-10 accuracy to 26.13% without altering any data. It treats the effect as sabotage, not as a measurable field.
- Semi-Cyclic Stochastic Gradient Descent shows that block-cyclic data arrival makes iterates cycle. Its fix, separate per-block averages, is proven only for convex losses.
- Neural Network Weights Do Not Converge to Stationary Points: An Invariant Measure Perspective shows late-training gradients stay large while loss is flat, so iterates settle into a stationary distribution rather than a point. It assumes default shuffling and leaves open what shapes that distribution.
Two papers constrain the design.
- Random Reshuffling is Not Always Better shows that at practical step sizes, reshuffling can lose to sampling with replacement, while reshuffling wins as $\eta\to 0$. Our expansion is a small-$\eta$ statement, so we must check where it breaks, and must not claim palindromes beat every order at large step sizes.
- Step Size Matters in Deep Learning and Non-Singularity of the Gradient Descent Map for Neural Networks with Piecewise Analytic Activations show that large steps alone create periodic orbits. That is a competing, order-independent explanation that the experiments must separate from ours.
Finally, the curriculum tension frames the payoff. Curriculum Learning for Natural Language Understanding reports gains with its "Cross Review" difficulty ranking. Do Data-based Curricula Work? reports that curricula fail on Transformers. Our decomposition makes a concrete prediction about which part of a curriculum produces each result.
Research Program
Estimating the curl. The first deliverable is a cheap estimator of the order curl in real networks. Run one epoch (or a short window of batches) forward from a checkpoint, then the same batches reversed from the same checkpoint. The difference in endpoints equals $2A_\pi + O(\eta^3)$. This needs only two extra training passes per measurement, with no Hessian products. We will check the $\eta^2$ scaling by halving $\eta$ and confirming the difference shrinks fourfold. That scaling check is also where the small-step regime visibly ends. Spot checks with Hessian-vector products on sampled batch pairs will confirm the bracket form.
Linking the curl to orbits. The second deliverable relates curl magnitude to orbit behaviour. We track three things:
- the Fourier spectrum of weight trajectories, following the variational-inference paper;
- the late-training gradient norm and loss plateau, following the invariant-measure paper;
- how far the weights move over a cycle compared with how much the loss changes over it.
The hypothesis is that persistent curl, not batch variance, predicts orbit amplitude and lost accuracy.
Palindromic schedules. The third deliverable is a family of palindromic schedules. Any curriculum or ordering is converted by running each period's order forward and then backward, keeping its pacing, step count and data multiset unchanged. For momentum and Adam the cancellation is only approximate. We will measure the residual curl directly rather than assume it vanishes, which matters given the momentum amplification seen in the Repeat-Until-Bored paper.
Baselines
Do-nothing reference. The do-nothing reference is plain random reshuffling at the same learning rate, schedule, batch size and step count. It is free, and the curriculum literature says it is hard to beat. The ordering-attack paper reports roughly 90% CIFAR-10 test accuracy for its reshuffled baseline, against 26.13% under the "Oscillation Outward" attack. Do Data-based Curricula Work? reports 18.3 BLEU for random order on WMT16 with T5, versus 17.0 for curricula.
For the palindromic fix, beating this reference does not mean exceeding random order. It means two things. First, a palindromized structured order should come back to within noise of random order where the original order fell short. Second, it should not lose the gains a curriculum had where that curriculum beat random order.
Strongest existing curriculum method. This is the Cross Review curriculum from Curriculum Learning for Natural Language Understanding, which ranks examples by agreement among teachers trained on data folds. It reports +0.9 average GLUE and +1.30 EM / +1.15 F1 on SQuAD 2.0 with BERT-Base over its baseline. We reimplement it with its published ranking procedure and the same BERT-Base fine-tuning budget, tuning learning rate on the same grid for every arm.
Beating it, or matching it as an explanation, means one result: the palindromized Cross Review schedule keeps the same gain. That would show the gain lives in pacing. Losing the gain under palindromization would instead show that rotation was helping, which our account does not predict and would count against it.
Iterate averaging. Polyak-style iterate averaging, and the per-block "pluralistic" averaging of the semi-cyclic SGD paper for block-cyclic data, are the natural competitors for removing order-induced drift after the fact. Each is applied to the same trajectories at no extra training cost. Palindromes beat them if they reach lower final loss, or higher accuracy, at equal steps on non-convex networks.
Key ablation. Forward versus reversed versus palindromic versions of the same batch sequence isolate the curl, because all three share identical batch contents and variance statistics.
Why not just…?
Why not just use random reshuffling and forget structured orders? For accuracy alone, this may be the right practical answer, and we do not claim palindromes will beat it on average. The proposal's value rests elsewhere. It explains why structured orders fail, which reshuffling cannot. It offers a way to keep curriculum pacing, the part that Curriculum Learning for Natural Language Understanding shows can help, without the rotation. It also covers settings where order is imposed rather than chosen, such as diurnal block arrival or an adversary controlling the sequence. If palindromized curricula never differ from random order, the result is still informative: curricula would carry no benefit beyond their rotation.
Why not just lower the learning rate? The curl scales as $\eta^2$, so shrinking $\eta$ suppresses it, and Random Reshuffling is Not Always Better shows small steps make reshuffling's pathologies vanish. This is a real threat. The cost is slower training, and the invariant-measure paper reports that smaller learning rates can yield larger gradient norms and sharper solutions. We settle the question by comparing palindromes at the original $\eta$ against forward orders with $\eta$ reduced until accuracy recovers, at matched step counts and matched wall-clock time.
Why not just call it gradient variance or forgetting? A variance or forgetting story predicts that an order and its reverse do equally well, because their batch contents and per-step variance are identical. Our account predicts opposite-sign rotation, and possibly different outcomes, for an order and its reverse. It also predicts that the palindrome repairs the damage while a variance-matched random order does not. The forward/reverse/palindrome ablation is the direct test.
Why not just average the iterates? The averaging guarantees in Semi-Cyclic Stochastic Gradient Descent rely on convexity. In a non-convex orbit, the average of points along a loop can sit off the low-loss set. We run averaging head to head and report both the averaged model's loss and its distance from the orbit.
Proposed Experiments and Decisive Checks
Toy problems. Start with Munro's parity and encoder tasks and the synthetic matrix-recovery setting of the reshuffling paper. There we can compute $A_\pi$ exactly and check two things: whether dead zones line up with persistent curl, and whether palindromic repeat-until-bored, alternating the pattern sequence direction, removes them with and without momentum.
CIFAR-10 ordering attacks. Next, reproduce the loss-sorted oscillation orders from the ordering-attack paper on CIFAR-10 with ResNet-18 and SGD. The decisive check is whether the palindromized attack order recovers most of the gap from 26.13% back toward roughly 90%, while the reversed attack order damages accuracy comparably to the forward one. If palindromes do not help here, the curl is not the attack's main mechanism and the account fails on its strongest case. We also report Fourier spectra of the weights and late-training gradient norms for each order.
Language tasks. Third, run GLUE and SQuAD 2.0 with BERT-Base under Cross Review and its palindrome. Then run WMT16 with T5 under the curricula from the data-curricula paper and their palindromes, with three or more seeds and matched tuning budgets. The prediction is that GLUE gains survive palindromization and the WMT16 deficit shrinks. For MLM pretraining we have no reported numbers beyond the qualitative slowdown in Do Data-based Curricula Work?, so we report time-to-loss-threshold against random order.
Validity check. Across all settings, the reverse-difference estimate must show the $\eta^2$ scaling before any causal claim is drawn from it. Where it does not, typically at large steps where the step-size-driven orbits of Step Size Matters in Deep Learning dominate, we report the account as out of scope rather than stretching it.
References
- Iterated Vector Fields and Conservatism, with Applications to Federated Learning
- Stochastic gradient descent performs variational inference, converges to limit cycles for deep networks
- Repeat Until Bored: A Pattern Selection Strategy
- Manipulating SGD with Data Ordering Attacks
- Semi-Cyclic Stochastic Gradient Descent
- Neural Network Weights Do Not Converge to Stationary Points: An Invariant Measure Perspective
- Random Reshuffling is Not Always Better
- Step Size Matters in Deep Learning
- Non-Singularity of the Gradient Descent Map for Neural Networks with Piecewise Analytic Activations
- Curriculum Learning for Natural Language Understanding
- Do Data-based Curricula Work?