A common way to find anomalies in video is to train a network to rebuild normal footage and then flag whatever it rebuilds badly. This only works if the network cannot simply copy its input. Otherwise it rebuilds anomalies just as well as normal events, and the error signal disappears. The field has two partial answers to this problem, and they point in opposite directions. The first gives the reconstructor long-range context with temporal self-attention, so it can tell that an event is out of place given what happens seconds earlier or later. The catch is that attention is a near-perfect copying device: a query token can attend to itself or its immediate neighbours and return its own content. The second answer blocks copying by hiding the centre of a convolution kernel, so each location must be inferred from its surroundings. The catch here is that the surroundings are only a few pixels and a few frames wide.
We propose a single attention block that does both jobs. The block attends along rows, columns, and time across a whole clip, but it is "hollow": each query is forbidden from attending to a small space-time window around itself. It must therefore reconstruct its own features entirely from distant context. Our central claim is that long-range attention improves reconstruction-based video anomaly detection only when the local neighbourhood is excluded. The same attention without the hole should reintroduce the copy path and help little or even hurt. Hollow attention should also catch anomalies that are defined by distant context, which local masked convolutions structurally cannot see.
Core Claim and What Is New
Two masked blocks are the direct ancestors of this idea. SSPCAB zeroes the centre of a dilated convolution ("donut kernel") and adds a loss that predicts the missing centre from its neighbours. It raised Avenue region-level detection (RBDC) from 41% to 62% and raised DRÆM's MVTec localization AP from 68.4 to 69.9. SSMCTB extends the mask to 3D and adds self-attention, but that attention runs only across channels. Its context is still a dilated local kernel, and it reaches 93.2% frame-level micro AUC on Avenue. Both blocks supply our method's principle: hide the query, predict it from context. We keep that principle and replace the bounded kernel with attention whose reach spans the whole frame and clip.
On the attention side, STATE adds temporal self-attention to a convolutional autoencoder, reporting 90.3% AUROC on Avenue and 77.8% on ShanghaiTech. It fights over-generalization in two ways. It crops tracked objects with Cascade R-CNN, and it nudges inputs at test time, which alone moved ShanghaiTech from 73.7% to 77.8%. Neither remedy is architectural. Its attention has no mask, and object crops blind it to scene-level anomalies. For efficiency, we borrow the factorizations from CCNet, which routes attention along rows and columns and reaches every pixel after two passes at a fraction of non-local memory, and from ViViT, which separates spatial from temporal attention. Neither paper was built for reconstruction, and neither has any reason to exclude the query from its own context.
The block works on a feature map of shape $T \times H \times W$. For a query at position $(t,y,x)$, it attends only to keys that share its row, its column, or its spatial location across time. On top of this, it applies an additive mask
$$ M_{(t,y,x),(t',y',x')} = -\infty \ \text{ if } |t-t'| \le r_t \text{ and } |y-y'| \le r_s \text{ and } |x-x'| \le r_s, \ \text{ else } 0, $$
where $r_t$ and $r_s$ are the temporal and spatial radii of the hole. The block output is trained with an auxiliary Huber loss to match the unmasked input feature at that position, as in SSMCTB. At test time, the block's own prediction error at each location becomes an extra anomaly map. The block drops into any reconstruction or prediction network in the same way SSPCAB does. When the hole shrinks to zero, it becomes ordinary axial attention, which gives us the key ablation.
Research Program
We will build three versions of the block and insert each into the same host network. The spatial-only version is a hollow criss-cross attention for images. The temporal-only version attends along the time axis at each location, with a temporal hole. The full version alternates the two. For video, the host is the frame-level framework into which SSMCTB was inserted for its Avenue and ShanghaiTech results; we reuse its training protocol unchanged, so the only variable is the block. For images, the host is DRÆM's reconstructive sub-network, which makes the MVTec comparison directly against the published 68.4 AP and 69.9 AP numbers.
The analysis centres on the hole radius. We will sweep $r_t$ from 0 (plain attention) upward and $r_s$ from 0 to a few feature cells, measuring two things at each setting. The first is the gap between the block's prediction error on anomalous regions and on normal regions. The second is how much of the attention mass lands within the local window when there is no hole. If the copy-path explanation is right, the unmasked block will concentrate its attention on or near the query and show a small anomaly–normal gap. As the hole widens, the gap should grow and then fall again once the hole starts removing context the prediction genuinely needs.
We will also test whether reach matters, separately from masking. To do this, we cap the attention span at a local window equal to SSMCTB's dilated receptive field. If this capped hollow attention matches the full version, the gains come from attention-as-aggregator rather than from long range.
Baselines
The naive reference is the host network with no added block, trained and evaluated under the identical protocol. Every gain is reported relative to this, because absolute AUCs on these benchmarks vary widely across host frameworks.
The strongest directly comparable competitor is SSMCTB's 3D block in the same host, which reports 93.2% frame-level micro AUC on Avenue in SSMCTB. It is the right rival because it shares our anti-copy principle and differs only in having local rather than global context. We will rerun it rather than quote it, using the same seeds, number of epochs, and hyperparameter search budget over loss weight and dilation as our block. Beating it means a consistent gain over three seeds on Avenue and ShanghaiTech in both frame-level AUC and RBDC, at comparable added latency. SSMCTB reports at most 0.4 ms of added inference time, and we report ours alongside.
The key ablation is the same factorized attention with no hole ($r_t = r_s = 0$), trained with the same auxiliary loss. This isolates the mask as the active ingredient. If it matches hollow attention, our central claim is false.
As external reference points under their own pipelines, we quote STATE's 90.3% AUROC on Avenue and 77.8% on ShanghaiTech from STATE. We also quote FPDM's 78.6% on ShanghaiTech from FPDM, a diffusion model that predicts the next frame's features from past frames. These are not controlled comparisons, because they use different hosts and, in STATE's case, object detection and optical flow. We state this plainly rather than claim to beat them.
On MVTec, the baselines are DRÆM at 68.4 pixel AP and DRÆM+SSPCAB at 69.9 pixel AP from SSPCAB. A spatial hollow block counts as winning if it clearly exceeds 69.9 AP in the same retraining protocol. For UBnormal, no reported result was found on Lacuna for a comparable unsupervised reconstruction baseline.
Why not just…?
Why not just increase the dilation of SSPCAB/SSMCTB's masked kernel? This is the most serious threat. A large-dilation donut kernel also excludes the centre and reaches farther, at almost no cost. However, a dilated kernel samples a fixed sparse grid, so its reach grows only by skipping everything in between. It also cannot reach seconds away in time without skipping most frames. SSMCTB itself reports high sensitivity to the dilation factor. We will run SSMCTB with dilation matched to our hole-plus-span geometry. If it equals hollow attention on the standard benchmarks and on the long-range diagnostic below, we would conclude that masking plus reach, not attention, is the contribution, and we would report it that way.
Why not just use temporal attention with test-time input nudging, as STATE does? STATE's own ablation shows that nudging is doing much of the work on ShanghaiTech: removing it drops the score from 77.8% to 73.7%. Nudging also costs several gradient steps per frame at inference. Nudging is complementary to our block, not a substitute for it, so we will run the unmasked-attention ablation with and without nudging. If nudging on the unmasked model closes the gap to the hollow model, the architectural mask is unnecessary.
Why not just mask input tubes, in the style of a masked autoencoder, instead of masking inside attention? Input masking also blocks copying. The difference is at inference: it reconstructs only the hidden tokens, so scoring every location requires many masked passes. Our internal mask scores every location in one pass. We will include a tube-masking variant with a matched compute budget and report accuracy per millisecond. If it wins at equal cost, the internal-mask idea loses its practical justification.
Why not just use a recurrent or memory-based model for long-range context? The file's evidence argues against this. Memory-augmented MemAE/MNAD models score around 50% AUC on context-dependent anomalies in the long-term Wrigley Field data in Trinity, and recurrent tracking drifts over long periodic sequences, as shown in Ultrasound Video Transformers. We will still include a ConvLSTM bottleneck in the host as a cheap reference.
Proposed Experiments
The main evaluation covers Avenue and ShanghaiTech, using frame-level micro and macro AUC plus region- and track-based detection criteria (RBDC and TBDC), under the controlled baselines above. MVTec AD serves as the image-only test of the spatial hollow block.
The decisive check is a diagnostic built for long-range anomalies. From normal test footage of Avenue and ShanghaiTech, we will splice in segments taken from elsewhere in the same scene. The result is locally plausible but globally inconsistent: people who reappear at a different location, abrupt reversal of an entire walking segment, or a frozen background over several seconds. Each manipulation is longer than any 3D kernel's temporal window. On this set, local masked convolutions should perform near chance, and hollow attention should not. We will report performance as a function of manipulation length, expecting a crossover near SSMCTB's temporal receptive field. If available, we will also evaluate on the Wrigley Field contextual split used in Trinity, which reports 92.0% on pseudo-contextual anomalies.
The claim is supported if three results hold together. Hollow attention beats the unmasked ablation. It beats matched-dilation SSMCTB, with the advantage growing with manipulation length. Its added latency stays within a few milliseconds. The claim is refuted if the unmasked ablation matches it, or if large-dilation masked convolution matches it on the long-range diagnostic.
References
- Self-Supervised Predictive Convolutional Attentive Block for Anomaly Detection
- Self-Supervised Masked Convolutional Transformer Block for Anomaly Detection
- Making Reconstruction-based Method Great Again for Video Anomaly Detection
- CCNet: Criss-Cross Attention for Semantic Segmentation
- ViViT: A Video Vision Transformer
- Feature Prediction Diffusion Model for Video Anomaly Detection
- Context-aware Video Anomaly Detection in Long-Term Datasets
- Ultrasound Video Transformers for Cardiac Ejection Fraction Estimation