Action recognition models trained on ordinary camera footage tend to fail when they are shown cartoons, game footage or other stylized video. The image literature attributes this failure to texture. Networks learn surface patterns such as fur, skin and fabric instead of shape, and drawings do not contain those patterns. Stylized video adds a second, temporal difference that this explanation leaves out. Animation is often drawn "on twos," where each drawing is held for two or more frames, so its rhythm is irregular, with holds, jumps and smears that real footage never shows. We propose to train video models with a single augmentation that randomizes both sides of depiction. It scrambles textures in a way that stays constant inside each short clip, so motion survives intact. It also resamples the clip's timing stochastically, using random holds, skips and irregular intervals. A consistency loss then makes the model give the same answer for differently "depicted" clips of the same video.
The case for doing this now comes from a benchmark result. On the VUDG video domain-generalization benchmark, cartoon, game and virtual styles cause large accuracy drops even for large video-language models: VideoLLaMA2-7B loses 15.4 percentage points under single-source shift. The same study reports that reasoning about temporal order degrades more than recognizing what is in a single frame. If that holds, image-style fixes that act only on texture are aimed at half the problem.
Core Claim
The non-photorealistic video gap has two separable components. The first is a texture component, which image methods already address. The second is a cadence component, which stochastic temporal sampling, a technique used so far against timestamp and clip-length shortcuts, can simulate directly. We claim that an augmentation combining the two closes more of the gap than either one alone, provided it follows two rules. First, the random texture transform must be shared across all frames of a snippet, meaning one short contiguous run of frames, and resampled only between snippets. Second, the cadence perturbations must mimic animation practice, with frame holds, irregular strides and occasional drops, rather than uniform jitter.
The contrast with prior work is direct. RandConv scrambles textures by convolving each image with a freshly drawn random filter, which forces the network to use shape. It is image-only and has no notion of time. SSA²lign samples random 8-frame snippets and enforces consistent predictions across snippets of one video. It applies no texture or cadence perturbation and is tested only on shifts between real-camera datasets. That paper also warns that frame-wise spatial augmentation causes "temporal rupture," meaning the destruction of the frame-to-frame motion cues the model needs. This warning is the constraint that forces our design to share the filter within a snippet.
Delta Sampling and FAR jitter frame intervals and starting points to stop models from keying on clip length or phase offset, but neither connects jitter to depiction style. Our proposal reuses SSA²lign's snippet consistency and RandConv's filters as components. What is new is the shared-within-snippet scrambling, the animation-cadence sampler, and the claim that the two ingredients target different parts of the gap.
Research Program
For each training video we draw $M$ snippets. Snippet $m$ receives a random start time, a random stride sequence and a random hold pattern. The hold pattern is a per-frame repeat count drawn from {1, 2, 3}, weighted toward 2 to match drawing on twos. Each snippet also receives a single random convolution filter $W_m$ with a random kernel size, applied identically to every frame of that snippet and blended with the original frame at a random mixing weight. The model $f$ is trained with
$$\mathcal{L} = \frac{1}{M}\sum_{m=1}^{M} \mathrm{CE}\big(f(x_m), y\big) + \lambda \frac{1}{M}\sum_{m=1}^{M} \mathrm{KL}\big(\bar p \,\|\, f(x_m)\big),$$
where $x_m$ is the augmented snippet, $y$ is the action label, CE is cross-entropy, $\bar p$ is the mean predicted class distribution over the $M$ snippets, and $\lambda$ weights the consistency term. The KL consistency term follows RandConv's multi-view form. Because the snippets differ in texture, timing and temporal position at once, the model has only one signal that agrees across all of them: the motion and shape of the action itself.
The work has three parts. The first builds the augmentation and trains a 3D CNN and TimeSformer on a photorealistic source. We include TimeSformer because Recur, Attend or Convolve? found that attention models and 3D CNNs latch onto static texture more than ConvLSTM does. The second is a measurement step. We build a controlled test bed by re-rendering held-out test videos in two ways. The texture-only version uses edge maps and line drawings with styles never seen in training, keeping the original timing. The cadence-only version keeps photographic frames but imposes animation-style holds and drops. Applying each factor separately lets us attribute the stylized-video gap to texture or to cadence. The third part tests on genuinely stylized video, namely VUDG's Cartoon, Game and Virtual domains, to check that the controlled findings hold outside the re-rendered test bed.
Baselines
The naive reference is a standard pipeline with uniform segment sampling, using the middle frame per segment in the style of TSN, and the usual crop and flip augmentation. The baseline is trained on the same source data with the same backbone and schedule. Every gain is reported relative to it.
The strongest existing temporal-sampling competitor is SSA²lign's stochastic snippet sampling with cross-snippet consistency, used without any of its few-shot target alignment terms. This isolates the sampling idea and gives it the same consistency loss we use. If our method beats it, the gain comes from texture and cadence randomization rather than from multi-snippet training.
The strongest motion-focused sampler is MGSampler, which places frames where image differences are large. It reports 39.5% top-1 on Diving48 in its own setting. Recur, Attend or Convolve? reports about 38.3% for ConvLSTM on Diving48 together with smaller drops on the texture-masked variant. That makes ConvLSTM the architectural alternative: rather than changing the data, choose a backbone with less texture bias. We train it on the same data so the comparison is fair.
STDN is the strongest video domain-generalization method in the file. It reports a +2.2% gain over VideoDG on HMDB→UCF, but the file gives no absolute value. We rerun it on our splits.
All methods receive the same source data, the same number of training clips seen, the same backbone where the method allows, and an equal hyperparameter search of 12 configurations. Model selection uses source-domain validation only. Our method beats the field if it improves average accuracy across the stylized target domains by a margin larger than the seed-to-seed spread over 3 seeds, compared with the best of these baselines, while staying within one point of source-domain accuracy.
Three ablations isolate the key ingredients. The first draws filters independently per frame, which tests the temporal-rupture prediction. The second shares one filter across the whole clip and uses uniform sampling, which tests texture alone. The third uses cadence-only sampling without filters. The decisive comparison is full method versus clip-shared filters alone on the cadence-perturbed and real-animation test sets.
Why not just…?
Why not just apply RandConv or style transfer frame by frame? Drawing a new filter or style for every frame makes every pixel flicker, which corrupts exactly the motion signal action recognition depends on. This is the temporal rupture SSA²lign describes. Our per-frame ablation measures the cost directly. We predict it lands below the naive baseline on Diving48, where motion is nearly the only cue. If it does not, the rupture concern is overstated for these backbones, and we will report that.
Why not just share one random filter across the whole clip and keep normal sampling? This is the real threat. It is simpler, it avoids rupture, and it may capture most of the texture benefit. We do not argue it away; ablation two runs it. Our claim survives only if the full method wins specifically on cadence-perturbed and real animated video, while the two are roughly tied on the texture-only test. If clip-shared filters match us on VUDG cartoons too, the honest conclusion is that cadence is a minor factor, and the paper becomes a measurement result about where the gap comes from.
Why not just pretrain on synthetic or massive data? SynAPT shows that synthetic pretraining helps on motion-heavy tasks, reaching 63.50% versus 62.84% on Diving48 for TSN. That gain is small, and its synthetic renders are realistic 3D with regular timing, so they do not teach irregular cadence. Massive real-data pretraining does not prevent large drops on stylized domains, as VUDG's VideoLLaMA2-7B and GPT-4o results show. We will also apply our augmentation on top of a SynAPT-initialized model to test whether the two effects add up.
Why not just use domain adaptation such as DANN? Domain adaptation assumes access to unlabeled target video. We study generalization to styles never seen during training, and RoCoG-v2 shows that DANN-style adaptation leaves a large gap under compound shifts, about 36.2% for X3D between source-only and target-only training.
Proposed Experiments
The primary source is Diving48, chosen because it is texture-light and motion-defined. We test on its S1, S2 and T variants from Recur, Attend or Convolve?, on our edge-map and line-art re-renders with held-out styles, and on cadence-perturbed versions. A second track trains on Mini-SSV2, using the same re-rendering protocol. The external check is VUDG. The file gives no reported result for supervised action classifiers on VUDG; its only numbers are video-language model accuracies, such as Qwen2.5VL-7B at 72.1% zero-shot, and roughly 17–18% for non-LLM VideoQA models. We would therefore map VUDG's 37 activity labels onto a classification protocol, and as a secondary probe fine-tune a small video-language model's visual encoder with our augmentation and report change in QA accuracy.
Metrics are top-1 accuracy on each target, the gap between source and target accuracy, and VUDG's temporal-ordering question subset. That subset directly tests the claim that cadence randomization protects temporal reasoning in particular. Two results would settle the core claim. First, the full method should shrink the cadence-perturbed gap far more than clip-shared texture randomization does, while the two tie on texture-only shift. Second, the per-domain ranking on VUDG cartoons should follow the cadence-perturbed test bed more closely than the texture-only one. If both hold, stylized video's gap is partly temporal, and stochastic sampling is the right tool for that part. If neither holds, texture is the whole story, and the clip-shared-filter result alone is the paper.
References
- VUDG: A Dataset for Video Understanding Domain Generalization
- Robust and Generalizable Visual Representation Learning via Random Convolutions
- Augmenting and Aligning Snippets for Few-Shot Video Domain Adaptation
- Delta Sampling R-BERT for Limited Data and Low-Light Action Recognition
- FAR: Fourier Aerial Video Recognition
- Recur, Attend or Convolve? On Whether Temporal Modeling Matters for Cross-Domain Robustness in Action Recognition
- MGSampler: An Explainable Sampling Strategy for Video Action Recognition
- Diversifying Spatial-Temporal Perception for Video Domain Generalization
- How Transferable are Video Representations Based on Synthetic Data?
- Synthetic-to-Real Domain Adaptation for Action Recognition: A Dataset and Baseline Performances