When video is turned into discrete tokens, meaning integer codes from a learned codebook so that a language-model-style Transformer can predict video one token at a time, most of the tokens in each frame simply repeat the code at the same position in the previous frame. A smaller fraction change, and those carry the motion. This proposal asks what happens to the motion tokens when such models are trained with communication-compressed optimizers. These optimizers cut the data exchanged between GPUs by keeping only the strongest components of each update. Our hypothesis is that the "strongest components" are dominated by the easy, abundant copy tokens. Aggressive compression could then match the uncompressed training loss while quietly discarding the learning signal for motion, and with it the capabilities that depend on motion, such as imitating a demonstrated action from a prompt video.
The discrete setting is what makes this testable. Because tokens are categorical, "this token changed from the previous frame" is an exact, free label. It lets us split the loss, and the gradient, into a copy part and a motion part without any extra model. We propose to use that split three ways. First, we diagnose how much of the motion gradient each compressor keeps. Second, we test whether aggregate loss hides capability loss. Third, we build split-budget compression, which reserves part of the communication budget for the motion gradient. A simpler competitor, upweighting motion tokens in the loss, is treated as a first-class rival rather than a strawman.
Core Claim
Communication-efficient optimizers have been validated almost entirely on text. DeMo keeps a local momentum buffer on each worker, transforms it with a discrete cosine transform, and sends only the top-k coefficients. It reports 85x less transfer on OLMo-300M (7.5 MB vs. 637 MB per step) "with zero loss in accuracy." SEPARATE projects gradients onto a shared random low-rank subspace, motivated by top-heavy gradient spectra. Its authors explicitly leave vision and multimodal models open. DiLoCo synchronizes only every 30 steps, and its scaling laws are stated only for dense text Transformers.
These methods share one assumption: an update that preserves the dominant directions preserves what the model learns. Our claim is that in discrete autoregressive video models this assumption fails in a specific, predictable place. The dominant gradient directions come from copy tokens. Motion tokens are a loss minority whose gradient lies partly outside the retained top-k set or low-rank subspace. As a result, validation loss can match while motion-dependent abilities degrade.
The quantity we track is the motion retention ratio. Write the step's gradient as $g = g_{\text{copy}} + g_{\text{move}}$, where $g_{\text{move}}$ is the gradient of the loss restricted to tokens whose code differs from the previous frame at the same position. Write $C(\cdot)$ for the compress-then-decompress operator. Then
$$r_{\text{move}} = \frac{\langle C(g),\, g_{\text{move}} \rangle}{\|g_{\text{move}}\|^2},$$
with $r_{\text{copy}}$ defined the same way. The hypothesis predicts $r_{\text{move}} \ll r_{\text{copy}}$ under aggressive compression, and predicts that this gap, not the loss gap, tracks the loss of in-context imitation.
Research Program
The testbed follows VidIT. VidIT tokenizes 16-frame clips with a pretrained image VQ-GAN and trains LLaMA-style decoders on Ego4d and Kinetics-600 by next-token prediction. Prepending a demonstration clip then makes the model imitate the demonstrated action. This is the right capability to watch, because it depends on cross-frame dynamics rather than on copying.
We train at 300M and 700M parameters under five regimes: AdamW with full data-parallel synchronization; DeMo at several top-k settings; SEPARATE at several ranks; DiLoCo with H=30; and our split-budget variant.
The first stage is diagnostic and is worth running even if the hypothesis fails. During training we compute $g_{\text{copy}}$ and $g_{\text{move}}$ on a fixed probe batch every few hundred steps, using two backward passes on the probe only. On these we measure per-layer $r_{\text{move}}$ and $r_{\text{copy}}$, the share of total loss carried by changed tokens, and the spectral overlap between the two gradient parts. That overlap is the test of whether SEPARATE's top-heavy spectrum assumption holds for video, and whether motion lives in its tail.
Because VQ-GAN codes can flicker under camera shake without real motion, we also use a stricter label. Under it, a token counts as moving only if its code differs from every code in its 3×3 neighborhood in the previous frame. The flicker rate itself is reported.
The second stage is the method. Split-budget compression gives the compressor two inputs instead of one. The compressor is DeMo's DCT top-k or SEPARATE's projection. Each worker computes its local update from the copy loss and the motion loss separately. It compresses the motion part with a reserved fraction of the coefficient or rank budget and the copy part with the remainder, so total bytes stay fixed. Error feedback is kept per stream.
The extra backward pass is the obvious cost. We measure it as wall-clock time, following the warning of Agarwal et al. that compression compute can cancel bandwidth savings. We also test a cheaper approximation that splits only the last few layers.
Baselines
The naive reference is AdamW with full gradient all-reduce at matched tokens seen. This is the quality ceiling and the communication worst case. Every other run is judged against it on loss, motion-token loss, and imitation metrics.
The strongest existing competitor is DeMo. It reports the largest reduction in the file (85x per step at OLMo-300M with no accuracy loss) and claims extreme sparsification (k=2) can even beat AdamW-DDP. If any compressor survives video unchanged, it is this one. We run DeMo at bytes-per-step matched to our split-budget variant, on the same data order, seeds, and token budget, with an equal learning-rate and k sweep for both.
SEPARATE is the second compression baseline. It is the method whose spectral assumption our diagnostic directly tests, and it is run under the same matching. DiLoCo with H=30 is the infrequent-synchronization reference. Its pseudo-gradient is not truncated by top-k, so it serves as a control showing whether the damage is specific to truncating compressors.
There is no reported result found on Lacuna for any communication-compressed optimizer on discrete video models. The relevant capability numbers come from VidIT, which reports 38.5% probing accuracy on Something-Something v2 against 27.1% for LVM, and a roughly 7–8% semantic-accuracy gain from an in-class demonstration. We do not assume our 300M reproduction matches these values. Each compressed run is compared with our own AdamW run.
The key ablation separates allocation from bytes. It gives the same extra budget to the plain compressor, for example DeMo at a correspondingly larger k. This shows whether split-budget helps because of where the budget goes or merely because of how much is sent.
Beating DeMo means the following. At equal bytes per step and equal wall-clock overhead (within 10%), split-budget closes at least half of DeMo's gap to AdamW on the in-context imitation gain and on Something-Something v2 probing accuracy, with no worse aggregate loss.
Why not just…?
Why not just watch the training loss and pick the most aggressive compression that matches it? That is exactly the practice our hypothesis says is unsafe, and it is directly testable. Across all compressed runs we regress the change in imitation gain on the change in aggregate loss, and separately on $r_{\text{move}}$ and motion-token loss. If aggregate loss predicts capability as well as the motion measures do, the hypothesis is wrong. The diagnostic contribution then reduces to confirming that DeMo-style results transfer to video, which is still useful but not new.
Why not just upweight the changed tokens in the loss? This is the serious threat. It costs nothing in communication or compute. It may well raise $r_{\text{move}}$ by pushing motion gradients into the dominant directions the compressor keeps. We run it as a full competitor, with the weight tuned under the same budget, alone and combined with DeMo and SEPARATE. If reweighting matches split-budget, the conclusion is that the fix is a one-line loss change paired with the motion diagnostic, and we will report that rather than the more complex method. Split-budget earns its place only if reweighting hurts copy-token quality, such as background stability, which reweighting trades away and split-budget does not.
Why not just sparsify the codebook output head by syncing only active rows, as in Fast Parallel Training of Neural Language Models? That method syncs only the output-layer rows for words in the batch plus a frequent slice. It paid off because recurrent language models had huge vocabularies relative to model size. In a 300M–1B video Transformer, the head for a codebook of a few thousand to tens of thousands of codes is a small fraction of the parameters. A 16-frame clip contains hundreds of tokens per frame, so most codes are likely active in every batch. We measure the active-row fraction and the head's share of bytes to confirm or refute this. We expect the backbone, not the head, to dominate communication. BitDP, which keeps embeddings and heads at full precision for stability, further suggests the head is the wrong place to compress.
Why not just use standard 8-bit collectives, or larger batches? Eight-bit formats give a fixed 2–4x reduction. They are orthogonal to the question here, which concerns the 30–85x regime where truncation decides what is kept. Larger per-device batches are constrained by video sequence memory. DiLoCo's analysis also indicates that standard data-parallel training degrades beyond critical batch sizes. DiLoCo itself is in the experiment, so whether infrequent synchronization sidesteps the motion problem is measured rather than assumed.
Proposed Experiments
Pretraining uses Ego4d and Kinetics-600 tokenized as in VidIT. A held-out check on ViMix-14M clips tests whether the copy/motion ratio, and hence the effect, depends on the corpus.
Evaluation covers four measures:
- validation loss, split into copy-token and motion-token loss;
- linear-probe accuracy on Something-Something v2, whose direction-of-motion classes are the natural probe for motion features;
- VidIT's in-context protocol, scored as the semantic-accuracy gain from an in-class demonstration over no demonstration;
- bytes per step and wall-clock time per step on a bandwidth-limited multi-node setup.
The first decisive check is whether, at DeMo settings that match AdamW's aggregate loss within noise, the motion-token loss and imitation gain degrade measurably while copy-token loss does not. The second is whether per-layer $r_{\text{move}}$, measured early in training, predicts which compressor settings later lose imitation. If it does, the ratio becomes a cheap early warning for tuning compression on any discrete-token video model. The third is the three-way comparison between split-budget compression, loss reweighting, and the matched-bytes compressor. It settles whether the fix needs a new compressor or only a new loss weighting.
A null result on the first check, where loss and capability move together, would be reported as evidence that text-validated compressors transfer safely to discrete video. That is also a result practitioners currently lack.
References
- DeMo: Decoupled Momentum Optimization
- SEPARATE: A Simple Low-rank Projection for Gradient Compression in Modern Large-scale Model Training Process
- Communication-Efficient Language Model Training Scales Reliably and Robustly: Scaling Laws for DiLoCo
- Video In-context Learning: Autoregressive Transformers are Zero-Shot Video Imitators
- On the Utility of Gradient Compression in Distributed Training Systems
- Fast Parallel Training of Neural Language Models
- BitDP: Ultra-low-bit Communication for Data Parallelism in LLM Training