Video networks that split each 3D convolution into a spatial 2D filter followed by a short temporal 1D filter (so-called (2+1)D or pseudo-3D networks) are always trained with first-order optimizers such as SGD or AdamW. Second-order optimizers are a natural candidate for them. These methods rescale each gradient step using an estimate of the loss surface's curvature, and on images and text they reach target loss in roughly half the steps. The catch is that they invert curvature matrices, and in 16-bit arithmetic that inversion often blows up. We argue that video adds a specific, predictable cause of this blow-up that images do not have. The temporal filter sees three or so consecutive frames at once, and consecutive frames are nearly identical. The curvature matrix built from those inputs therefore has near-duplicate blocks and is close to singular.
Our proposal is to separate this "frame redundancy" part of the curvature into a tiny matrix, about 3×3 per layer, and to invert that matrix exactly in 32-bit precision. The large channel part of the curvature, which is well conditioned, can then go to an existing low-precision-stable second-order method. If the diagnosis is right, two things should follow. Instability of standard second-order methods on video should track the frame sampling rate, and the fix should remove that dependence at negligible cost. In two sentences: second-order optimizers break on video because neighbouring frames make the temporal filter's input covariance nearly singular. Isolating that redundancy in a small exactly-inverted factor should make these optimizers usable for video in reduced precision.
Core Claim
Take a temporal convolution with $k_t$ taps acting on $M$ intermediate channels. K-FAC is a second-order method that approximates each layer's curvature as a Kronecker product of an input-activation covariance and an output-gradient covariance. In K-FAC, the input factor is built from stacked patches $a = [x_{t-1}; x_t; x_{t+1}] \in \mathbb{R}^{k_t M}$, where $x_t$ is the $M$-channel feature at frame $t$. Its blocks are $E[x_{t+i} x_{t+j}^\top]$.
Two approximations simplify this. If features are roughly stationary in time, and if the frame-to-frame correlation is shared across channels, the factor becomes
$$A \approx R \otimes \Sigma,$$
where $R \in \mathbb{R}^{k_t \times k_t}$ is a Toeplitz matrix of tap-to-tap correlations $\rho_{|i-j|}$ and $\Sigma \in \mathbb{R}^{M \times M}$ is the per-frame channel covariance. The condition number of a Kronecker product is the product of the factors' condition numbers:
$$\kappa(A) \approx \kappa(R)\,\kappa(\Sigma).$$
As consecutive frames become more similar, $\rho \to 1$ and the smallest eigenvalue of $R$, which corresponds to the "difference between frames" direction, goes to zero. The overall conditioning then degrades in proportion, however benign $\Sigma$ is.
This makes three predictions. Instability concentrates in the temporal convolutions rather than the spatial ones. It worsens as the temporal sampling stride shrinks. It disappears if $R$ is handled separately and exactly.
The method follows directly from this structure. We split the temporal layer's activation factor three ways, into gradient covariance ⊗ $R$ ⊗ $\Sigma$. $R$ is estimated and inverted in FP32 with its own small damping, which costs a 3×3 solve per layer. $\Sigma$ and the gradient factor go to an inverse-free, BF16-stable update.
Relation to Prior Work
The low-precision fixes we build on all target ill-conditioning generically, without asking where it comes from:
- Structured Inverse-Free Natural Gradient Descent avoids inversion by updating curvature in a matrix-logarithm space. It remains stable in BF16 where K-FAC spikes, and it supports diagonal, block-diagonal and Toeplitz structure on the factors.
- MKOR updates inverse factors through rank-1 Sherman-Morrison steps in FP16.
- KrADagrad replaces inverse matrix roots with positive ones.
- SPlus rotates gradients into a cached eigenbasis and applies sign normalization.
All four were evaluated only on 2D vision models, Transformers or graph networks. SPlus's authors explicitly leave non-Transformer architectures open. These methods supply our machinery for the $\Sigma$ factor. Our contribution is to identify the video-specific source of ill-conditioning and route it around them.
Two papers constrain the design:
- Kronecker-Factored Approximate Curvature for Modern Neural Network Architectures shows how statistics over weight-shared positions should be aggregated. We follow its "expand" treatment, because the temporal convolution shares weights across positions in time, height and width, and its loss contribution is spread across those positions.
- Shampoo already keeps one preconditioner per tensor dimension, including a $k_t \times k_t$ one for the tap dimension of a temporal kernel. It is therefore the closest existing mechanism, and we treat it as a central competitor rather than a foil. Shampoo's per-dimension factors are built from gradients, whereas the redundancy we identify lives in the input activations. Whether that distinction matters in practice is an empirical question this proposal answers.
On the architecture side, A Closer Look at Spatiotemporal Convolutions for Action Recognition introduced R(2+1)D. It reports that factorization eases optimization under SGD. That finding motivates the skeptical view that video networks need no curvature information at all, and we test this directly. Learning Spatio-Temporal Representation with Pseudo-3D Residual Networks supplies the parallel and hybrid block variants. In those variants, the spatial and temporal filters read the same input, which lets us check whether the redundancy is specific to the temporal branch.
Research Program
The work has three parts.
Diagnostic study. We instrument R(2+1)D-18 and a P3D variant trained with K-FAC in FP32. Every few hundred steps we log the eigenspectra of each layer's activation factor, separately for spatial and temporal convolutions. We also measure how well $R \otimes \Sigma$ fits the true temporal factor, using the relative Frobenius error of a nearest-Kronecker-product fit. Finally, we vary the clip sampling stride across 1, 2, 4 and 8 frames. This tests whether the temporal factor's smallest eigenvalue moves as $1-\rho$ predicts, and whether the BF16 divergence events of plain K-FAC occur in temporal layers first.
Method. We implement tap-separated preconditioning on top of an existing SINGD implementation:
- Temporal layers: $R$ is estimated by a running average of the $k_t \times k_t$ correlation between taps, after normalizing per channel, and inverted in FP32. $\Sigma$ and the gradient factor use SINGD's inverse-free BF16 update.
- Spatial layers: these receive unmodified SINGD.
The extra memory is $k_t^2$ floats per temporal layer.
Extensions. We check whether the same split helps SPlus by applying the eigenbasis rotation only to $\Sigma$. We also check whether it carries over to a monolithic 3D convolution baseline, where the tap dimension is entangled with the spatial taps.
Baselines
The do-nothing reference is the standard recipe: R(2+1)D-18 trained with SGD with momentum, and separately AdamW, under BF16 mixed precision. Each uses a cosine schedule and the same hyperparameter sweep budget as every other method, namely an equal number of trials over learning rate and weight decay. This is the bar any second-order method must clear, because it is what practitioners actually run. For reported reference points, Tran et al. give R(2+1)D a Sports-1M top-1 of 57.0% versus 47.9% for P3D and 46.1% for C3D (Tran et al.). On Kinetics-400 they report only a relative +4.5% over I3D from scratch. The Kinetics dataset paper reports 56.1% top-1 for its 3D ConvNet (Kay et al.). No reported result was found on Lacuna for any second-order optimizer on any video benchmark. Our first-order runs therefore serve as the operative baseline numbers, and the published figures only sanity-check that our training recipe is in a reasonable range.
The second baseline is standard K-FAC with a tuned Tikhonov damping schedule. It receives the same sweep budget, with damping added as a swept dimension. This is the "just damp harder" competitor.
The strongest existing methods are SINGD (dense, and with block-diagonal and Toeplitz structure) and Shampoo or SPlus with per-dimension factors on the 5D kernel tensor. Shampoo and SPlus are the most dangerous competitors because they already isolate the tap dimension. All are run in BF16 with identical data pipelines, clip lengths, batch sizes and preconditioner update frequencies. Results are reported both per step and in wall-clock time on the same hardware.
Two ablations isolate the key ingredient:
- Precision ablation: tap-separated preconditioning with $R$ also kept in BF16 tests whether the precision assignment is what matters.
- Structure ablation: the same split with $R$ fixed to the identity tests whether explicitly modeling frame correlation matters or whether merely separating the tap dimension suffices.
Beating the field means two things together:
- Stability: zero divergences across the sweep at strides 1 and 2, where we expect plain K-FAC to diverge.
- Speed: reaching the AdamW baseline's final validation accuracy in at most roughly 75% of its wall-clock time. That threshold is the upper end of the step savings Eschenhagen et al. report for K-FAC over NAdamW. It is a fair target only if per-step overhead stays small.
Why not just…?
Why not just use AdamW, since factorization already makes video networks easy to train? Tran et al. do report that (2+1)D lowers training error under SGD, and this is a genuine threat. Even so, the step reductions of 44–58% reported for SPlus over Adam (Frans et al.) and 50–75% for K-FAC over NAdamW (Eschenhagen et al.) are large enough to matter for expensive video training. We do not assume those gains transfer. If our best second-order run fails to beat tuned AdamW in wall-clock time on Kinetics-400, we report that video factorization removes the need for curvature, which is itself a useful negative result.
Why not just increase the damping? Damping adds a multiple of the identity to every direction equally. Pushing it high enough to cover the nearly-singular frame-difference direction also flattens the informative channel directions toward plain SGD. Lin et al. show that K-FAC still spikes in BF16 even with damping. The damping-swept K-FAC baseline measures this directly. If some damping value matches our method at every stride, the proposal's mechanism is unnecessary.
Why not just use Shampoo or SPlus, which already give the tap dimension its own preconditioner? This is the closest alternative and may well work. The difference is that their tap factor comes from gradient statistics, while the redundancy we predict sits in the activations, so the two need not coincide. The stride sweep settles the question. If SPlus is already insensitive to stride and matches our accuracy, the conclusion is that per-dimension tensor optimizers are the right tool for video. We would report that, with our method serving as an explanation of why they work.
Why not just use SINGD's block-diagonal structure, which drops cross-tap blocks entirely? Dropping the off-diagonal blocks removes the near-duplication, but it also throws away the frame-difference signal, which is exactly what motion modelling needs. The $R=I$ ablation approximates this. A gap between it and the full method on motion-heavy classes would show that the correlation must be modelled, not discarded.
Proposed Experiments
Datasets and setup. The main testbed is Kinetics-400 (Kay et al.) with R(2+1)D-18 trained from scratch, using 16-frame clips at strides 1, 2, 4 and 8. UCF101 (Soomro et al.) and HMDB51 provide smaller from-scratch settings where we can run full sweeps with several seeds. A P3D-B/C variant tests whether the effect holds when spatial and temporal branches share inputs.
Metrics. We report top-1 and top-5 accuracy, the steps and wall-clock time needed to reach the AdamW baseline's final accuracy, and the fraction of sweep runs that diverge or produce invalid (NaN) values. We also report per-layer condition numbers of the factors and peak memory.
Decisive checks.
- Spectral prediction. In the FP32 diagnostic, the temporal factor's smallest eigenvalue should rise with stride, and $R \otimes \Sigma$ should fit the true factor with low error. If the fit is poor, the method's premise fails.
- Stride dependence. Plain K-FAC's BF16 divergence rate should fall with stride while the tap-separated method stays flat. If divergence does not depend on stride, frame redundancy is not the cause.
- Ablations. If the BF16-$R$ ablation matches the full method, precision assignment is irrelevant. If the $R=I$ ablation matches, explicit correlation modelling is irrelevant.
- Comparison with SPlus. Matched against SPlus at equal wall-clock time, this determines whether the contribution is a new optimizer or an explanation of an existing one.
References
- Structured Inverse-Free Natural Gradient Descent: Memory-Efficient & Numerically-Stable KFAC
- MKOR: Momentum-Enabled Kronecker-Factor-Based Optimizer Using Rank-1 Updates
- KrADagrad: Kronecker Approximation-Domination Gradient Preconditioned Stochastic Optimization
- A Stable Whitening Optimizer for Efficient Neural Network Training
- Kronecker-Factored Approximate Curvature for Modern Neural Network Architectures
- Shampoo: Preconditioned Stochastic Tensor Optimization
- A Closer Look at Spatiotemporal Convolutions for Action Recognition
- Learning Spatio-Temporal Representation with Pseudo-3D Residual Networks
- The Kinetics Human Action Video Dataset
- UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild