A Learned Boundary Stream from Raw Frame Differences for Temporal Action Detection

Temporal action detection means finding where each action starts and ends in a long, unedited video and naming it. The strongest detectors on the THUMOS'14 benchmark do not look at pixels at all. They read a precomputed sequence of feature vectors, one per 16-frame chunk. Those vectors come from a video classifier trained on Kinetics and then frozen, and the detectors usually add a second stream built from optical flow, a hand-designed motion estimate computed offline. This is the video version of the hand-built feature pipelines that NLP abandoned. The upstream features were designed to say what happens in a clip. They were not designed to say when something changes. End-to-end training fixes this, but it is expensive: the encoder must be backpropagated through at reduced resolution, and the best end-to-end system in our file still trails the best frozen-feature systems by a wide margin.

We propose a different split of the work. The frozen clip features keep supplying "what is happening." A small, separately learned network supplies "is something changing right now." This network reads raw, low-resolution frame differences at the native frame rate and is trained only through the detection loss. It is a stack of dilated one-dimensional convolutions, meaning convolutions over time whose gaps between taps double at each layer, so the time span they cover grows quickly while the cost stays linear in video length. It replaces the optical flow stream entirely. The bet is that the accuracy lost by freezing the encoder is mostly high-frequency boundary evidence, which a cheap convolutional pathway can recover. If so, flow and full end-to-end encoder training both become unnecessary for boundary precision.

Core Claim

The claim is that frozen chunk-level features lose mainly boundary timing, not semantics, and that timing can be learned from raw frame differences by a lightweight dilated temporal convolutional network (TCN) under detection supervision. Fused into a strong convolutional detection head, this stream should match or exceed the published two-stream frozen-feature result without computing optical flow. The gains should concentrate at strict overlap thresholds and on short actions, which are exactly where current detectors are weakest. The claim is falsifiable in four ways. It fails if the gain appears evenly across overlap thresholds. It fails if it survives temporally shuffling frames within each chunk. It fails if denser frozen features close the gap. It fails if fine-tuning the encoder end-to-end at matched memory does better.

Where This Sits

Three bodies of work shape the design.

The first motivates it. An Empirical Study of End-to-End Temporal Action Detection backpropagates detection losses into SlowFast and other encoders at 96×96 resolution. It reports up to 11% average mAP improvement over frozen features on THUMOS14 and 54.2 mAP with a TadTR head. Low-Fidelity End-to-End Video Encoder Pre-training finds that the detection supervision signal matters more to the encoder than visual fidelity. It shows this on ActivityNet and HACS, not THUMOS'14, and re-freezes the encoder for the final stage. Both papers treat the whole encoder as the thing to adapt. We ask whether a much narrower learned pathway captures most of the benefit, and which kind of information that pathway supplies.

The second supplies the method. TDN: Temporal Difference Networks shows that subtracting adjacent raw frames inside a 2D CNN is a cheap substitute for optical flow in trimmed-clip classification, reaching 52.3% top-1 on Something-Something V1. It was never tested on untrimmed boundary detection. Convolutional Sequence Modeling Revisited establishes residual dilated TCNs as trainable, parallel sequence models with long effective memory. Character-level Convolutional Networks for Text Classification is the conceptual template: 1D convolutions over the rawest available primitive in place of engineered units. It also carries a warning that such models needed millions of examples to beat engineered features, and THUMOS'14 has about 200 training videos. We take that warning seriously, as discussed below. It is why the learned stream is deliberately small and confined to a local, low-capacity job.

The third constrains the design. TriDet shows that self-attention in detectors such as ActionFormer makes features at different time steps collapse toward similarity with depth. Its convolutional layers and its head, which predicts a probability distribution over boundary positions, avoid this collapse and reach 69.3% average mAP on THUMOS14. We therefore build on TriDet's head rather than an attention head. TFE-DCN finds that dilated context helps only up to about three layers on THUMOS'14 before background noise hurts. Non-Local Temporal Difference Network reports strong THUMOS-14 numbers but notes persistent failure on very short actions. Both use frozen features, and TFE-DCN uses precomputed flow. Together they argue that one receptive field cannot serve both boundary sharpness and long context. Our design separates the two: the boundary stream gets a short, fine-grained receptive field, and long context stays with the frozen features and TriDet's pyramid.

Research Program

The boundary stream. Each video is decoded at native frame rate and resized to 64–96 pixels. Adjacent frames are subtracted, and a tiny shared 2D stem pools each difference image into a vector. That gives one vector per frame, thousands per video. A residual TCN with non-causal dilations of 1, 2, 4, and 8 runs over this sequence. Its receptive field is a few dozen frames, well under one second. Its output is pooled onto the frozen features' time grid and concatenated as a second stream, in place of flow, at the input of TriDet's feature pyramid. Only the stem, the TCN, and the detection head are trained. Memory therefore scales with a small per-frame vector rather than with a 3D encoder's activations, which should let full videos fit in one pass without the chunking or resolution cuts that limited the end-to-end study.

Data scarcity. Two measures address the char-CNN warning. First, the stream has a narrow job, detecting local visual change, rather than learning action semantics. Second, we test an optional pretraining stage on ActivityNet-v1.3 with a class-agnostic boundary objective before THUMOS'14 training.

Diagnosis. The analysis decomposes the known end-to-end gain into three candidate sources: denser temporal resolution, adapted appearance semantics, and learned motion or boundary evidence. We give each its own control, so the proposal produces an explanation of why end-to-end training helps, not only a number.

Baselines

Naive reference. TriDet on frozen RGB-only features with no motion stream. This is what removing flow costs if nothing replaces it, and any claimed gain is measured against it first.

Strongest existing method. TriDet with its standard frozen two-stream features. TriDet reports 69.3% average mAP on THUMOS14. We rerun it from its released configuration with the same seeds, epochs, and tuning budget as our model. The file does not state TriDet's exact averaging range, so we report both the reproduced number and the published one. We also rerun NTD, which reports 71.6% [email protected] and 66.8% average mAP. Its claim to beat ActionFormer by 6.0 at [email protected] sits awkwardly beside ActionFormer's own reported 71.0% [email protected], so we compare everything under one evaluation script. Beating TriDet means matching or exceeding the reproduced two-stream average mAP without flow, with a statistically clear gain at tIoU 0.6–0.7 across at least three seeds.

End-to-end competitor. The recipe of Liu et al., with its best reported THUMOS14 result of 54.2 mAP using TadTR, transplanted onto TriDet's head. The encoder is fine-tuned at 96×96 and given the same GPU memory and wall-clock budget as our model, and also run once at double budget.

Ablations that isolate the key ingredient. The same stream is fed raw frames instead of differences. The stream's weights are trained with a clip-classification loss and then frozen, which tests whether detection supervision is what matters. TDN's pretrained short-term difference features are used frozen. The TCN's dilations are swept to change its receptive field. Finally, the frames inside each chunk are temporally shuffled at test time.

Why not just…?

Why not just plug TriDet's head into the existing end-to-end pipeline? This is the obvious combination and a real threat, because it may simply work. We run it directly as the end-to-end competitor at matched memory. If it beats our stream at equal cost, our narrow-pathway hypothesis is wrong and we will say so. If it wins only at several times the memory, the result favors the cheap stream. Either outcome is informative.

Why not just extract frozen features at a denser stride? The file's evidence against frozen features concerns what they encode, not how often they are sampled, so a denser grid might recover much of the gain for free. We test this with a control that uses the same frozen extractor at a stride of 4 instead of 16. The prediction is that denser frozen features help coarse localization but not high-tIoU precision on short actions. That prediction is a real risk.

Why not just keep optical flow? Flow works, but it is offline, slow, and storage-heavy. The evidence that learned substitutes suffice comes from trimmed classification in TDN and from a different benchmark in LoFi, which reports 24.64% vs. 23.00% on HACS, not from THUMOS'14 boundaries. We therefore also run flow plus our stream together. If adding the stream to flow gives nothing, the stream is a flow substitute. If it adds on top of flow, it carries boundary information that flow does not.

Why not just fine-tune the last encoder stage? This is a cheaper end-to-end variant that adapts semantics. It is included as a control. It tests whether the gain comes from adapted appearance or from new timing evidence.

Decisive Experiments

The main benchmark is THUMOS'14, evaluated by mAP at tIoU 0.3 to 0.7 and their average. We add per-duration breakdowns (short, medium, long), boundary error in frames, peak training memory, throughput, and preprocessing time against flow extraction. EPIC-KITCHENS 100 serves as a dense, short-action stress test. ActivityNet-v1.3 serves as a long-action check where we expect little gain.

Four results would settle the claim. First, the flow-free stream must match two-stream TriDet. Second, its advantage over RGB-only must grow with the tIoU threshold and be largest on short actions. Third, within-chunk shuffling must erase that advantage, which would show the stream uses timing rather than appearance. Fourth, neither denser frozen features nor last-stage fine-tuning should close the gap at equal cost.

If the first check passes but the third fails, the stream is only adding appearance capacity, and the timing story is wrong. If the end-to-end TriDet transplant dominates at matched memory, the conclusion is that full encoder adaptation, not a side stream, is the right fix. That is still a clean answer to an open question.