Locality as a Substitute for Augmentation in Small-Data Transformers

Vision transformers trained on small datasets lean heavily on hand-designed data augmentation, meaning random crops, color shifts, and image mixing. Take those away and accuracy collapses. This note tests a specific explanation: much of what augmentation buys a small-data transformer is compensation for a missing preference for nearby inputs. A convolutional network gets that preference for free. A transformer has to learn it, and learning it takes data.

If the explanation holds, a soft, learnable locality bias on attention should close most of the gap between "with augmentation" and "without augmentation." It should do so in modalities where no trustworthy augmentation exists, such as speech spectrograms and wearable-sensor signals. It should also make attention cheaper, because heads that stay local after training can be frozen into fixed, banded patterns that never compute query-key products. One mechanism would then address three problems at once: augmentation dependence, data hunger, and attention cost.

Core Claim

Each attention head gets a distance penalty on its attention scores, with a learned width. The widths start narrow, so every head initially attends only to neighbouring tokens. Gradient descent is free to widen them. In symbols, head $h$ computes attention scores $q_i^\top k_j/\sqrt{d} - \lVert p_i - p_j\rVert^2 / (2\sigma_h^2)$. Here $q_i, k_j$ are the query and key vectors, $d$ is their dimension, $p_i, p_j$ are token positions on the input grid (1D for sensors, 2D for images and spectrograms), and $\sigma_h$ is a learned width initialized to about one token.

After training, any head whose $\sigma_h$ stays below a threshold is replaced by a fixed Gaussian band with no query or key projection. It is executed as a dense windowed operation. The remaining heads stay fully learned and global.

The central measurable quantity is the augmentation-dependence gap: accuracy with the modality's standard hand-crafted augmentation minus accuracy with none, at matched data, compute, and tuning. We claim the locality bias shrinks this gap by at least half relative to a plain transformer. We also claim that, unlike architectures with permanently hard-wired convolutions, it stops constraining the model as the dataset grows, because the widths expand.

We do not claim that a learnable distance bias on attention is new in the broader literature. What is new here is threefold. First, the gap is used as the target of study. Second, the test extends to non-visual modalities where augmentation is unreliable. Third, locality is turned into an inference-time compute saving.

Grounding in Prior Work

Two results motivate the hypothesis.

Hard-Coded Gaussian Attention replaced learned self-attention in translation with fixed Gaussian bands. It matched the learned baseline (30.3 vs 30.0 BLEU on IWSLT16 En-De) and raised throughput by 30.2%. The paper's rationale is that learned heads drift toward local bands anyway.

MSCViT showed that injecting convolutional locality into a small ViT lifts CIFAR-100 accuracy from the 69.62% of PVT-T to 84.68%.

Two findings constrain the design.

The same Gaussian paper found that fixed local patterns degrade on long-distance dependencies, measured with the LingEval97 probes. So locality cannot be permanent for all heads.

The research file notes, as a known limitation rather than an MSCViT result, that heavily hard-wired convolutional and wavelet designs like MSCViT are expected to saturate when scaled to larger data. So the bias must be able to fade.

The learned width is our answer to both constraints. Heads that need long range can widen. Heads that do not can be hard-coded and made cheap.

Our augmentation angle differs from the modality-agnostic augmentation line. MODALS moves augmentation into latent space. Viewmaker Networks learns perturbations adversarially, and it reports +9 points on speech and +17 on Pamap2 over hand-designed views. Both methods keep augmentation and try to make it universal. We ask instead how much augmentation is needed once the architecture has the right prior. MODALS also reports that latent augmentation fails when early representations are poorly separated, which is exactly the small-data cold start. A locality prior might repair that failure, so we also test whether the two combine.

On the cost side, CCNet factorizes dense attention along rows and columns. It cuts memory 11x relative to non-local blocks, at the price of axis-aligned routing. Separately, the file cites Xu et al. for the point that irregular sparse indexing costs latency on accelerators. Our converted heads are contiguous bands, which avoids that problem. We will report measured wall-clock time rather than FLOPs alone.

Research Program

We build one transformer family with the learnable distance bias and train it from scratch on four settings. The two image settings are CIFAR-100 and TinyImageNet. The two non-image settings are Google Commands and VoxCeleb1 speech, fed as spectrogram patches, and Pamap2 activity recognition, fed as 1D windows. In a scaling track we train on 10%, 30%, and 100% of ImageNet-1K.

Every model is trained under three augmentation conditions. The first uses no augmentation. The second uses the modality's standard hand-crafted recipe. The third uses MODALS-style latent augmentation, which serves as the modality-agnostic option.

Three analyses follow from the trained models. First, we record each head's width trajectory over training and across data scales; the prediction is that final widths grow with dataset size. Second, we measure the accuracy lost, and the latency and memory gained, when narrow heads are frozen into fixed bands, sweeping the threshold. Third, we probe long-range behaviour. For images, the probe is a synthetic task that requires relating distant patches. For Pamap2, it is classification from windows whose discriminative events are far apart. The probes check whether the widened heads actually carry long-range information, which is the documented failure of fully hard-coded attention.

Baselines

The naive reference is a plain ViT of matched parameter count and FLOPs. It uses standard learned positional embeddings and is trained on identical data under the same three augmentation conditions. This model defines the gap we aim to close.

The strongest existing method on the small-image benchmarks is MSCViT. The file reports 84.68% top-1 on CIFAR-100 for MSCViT-S (14.0M parameters, 2.5 GFLOPs) and 72.11% on TinyImageNet (MSCViT). We will rerun it with its released recipe and also under our no-augmentation and latent-augmentation conditions, since its behaviour without augmentation is not reported. Beating it means one of two things. Either we match its augmented accuracy within one point at equal or lower FLOPs while showing a smaller augmentation gap, or we overtake it on the 100% ImageNet-1K track, where we expect its fixed convolutions to plateau.

For the ImageNet scaling track, the reference is DeiT-Ti. The file reports 72.2% on ImageNet-1K for DeiT-Ti, and 74.0% with self-supervised teaching-assistant distillation (Self-supervised Models are Good Teaching Assistants).

For the non-image tasks, the file gives only relative gains for Viewmaker, so there is no absolute number to beat. No reported result was found on Lacuna for a from-scratch supervised transformer on Google Commands, VoxCeleb1, or Pamap2. We will establish these references ourselves by running the plain ViT and a convolutional-hybrid transformer.

Three ablations isolate the key ingredient, the learned and expandable width. The first fixes the width at its initial value, which is the hard-coded Gaussian design ported to vision and signals. The second uses permanent local windows. The third is a convolutional hybrid that puts depth-wise convolutions in the feed-forward block, as ConvFormer does, matched in parameters. All models share optimizer, epoch budget, and an equal hyperparameter search budget of the same number of trials per method.

Why not just…?

Why not just use stronger hand-crafted augmentation? On images this is a real competitor, and our augmented condition measures it directly. The claim is not that augmentation stops helping. The claim is that the locality bias makes it less necessary. The argument bites hardest where augmentation is untrustworthy. JNDMix shows that visible distortions corrupt labels in quality-assessment tasks. The file's summary of Extending the WILDS Benchmark reports that adaptation methods fail on non-visual shifts where label-preserving transformations are lacking; the file gives no number for this. If heavy augmentation plus a plain ViT matches our no-augmentation model on speech and sensor data, the hypothesis is wrong.

Why not just add convolutions, as MSCViT and ConvFormer do? This is the most serious threat. A convolutional hybrid may match us on small-data accuracy, and we say so openly. The distinction must show up elsewhere. First, the scaling track should favour our model at full ImageNet-1K, where a hard-wired local prior cannot be switched off. Second, our model should deliver an inference-time attention saving from converted heads, which a convolution adds cost rather than removes. If the convolutional hybrid wins on both accuracy and scaling, the contribution shrinks to the cost result alone, and we will report it that way.

Why not just use a large-kernel CNN like ParCNetV2? ParCNetV2 reaches 84.6% on ImageNet-1K with global oversized kernels. However, it depends on custom low-level kernel implementations, and its content-independent weights cannot route information by content. We include it only on the image benchmarks, as an upper reference, and do not claim to beat it there. Our question concerns transformers, whose content-dependent routing is what the long-range probes test.

Why not just pre-train with masked autoencoding? Masked pre-training helps small-data transformers (TransVLAD). It is slow, though: ccMIM needs 300 epochs just to match a 1,600-epoch MAE. It also presumes a large unlabeled pool, which is often absent in the sensor and medical settings motivating this work. Pre-training is complementary to our approach, so we treat it as orthogonal and do not compete with it.

Decisive Checks

The proposal succeeds if three results hold.

The first and most important concerns the augmentation gap on the non-image tasks. The learned-width model must close at least half of the plain ViT's gap on Google Commands and Pamap2, and must beat the fixed-width ablation there. The fixed-width comparison separates "locality helps" from "adjustable locality helps."

The second concerns scaling. Learned widths must grow measurably from the 10% to the 100% ImageNet-1K subset, and the model's advantage over the convolutional hybrid must widen with data rather than shrink.

The third concerns cost. Converting narrow heads must cost under half a point of accuracy while giving measured, not merely theoretical, reductions in latency and memory. The long-range probes must show that the heads which stay learned retain distant-dependency accuracy, unlike the fully hard-coded variant.

If only the image results hold, the idea reduces to another small-data ViT. The non-image gap result is what the proposal is betting on.