When two actions happen at once in a video, such as a person talking on the phone while cooking, today's best detectors of overlapping actions see only one feature vector per moment. That vector is shared by every action label. The feature extractor has already averaged each frame over its whole area. So the small, hand-sized evidence for "holding a phone" and the whole-body evidence for "standing at the stove" are blended before the temporal model ever sees them. This proposal keeps a small multi-scale spatial map for every time step. Each action class then reads that map with its own learned query, so different concurrent actions get different frame summaries drawn from different regions and scales. The claim is simple and testable. Much of what makes overlapping actions hard to separate is not temporal at all. It comes from this early spatial averaging. A per-class, scale-aware readout should recover accuracy exactly on the frames where several actions co-occur.
The opportunity matters because the field has worked hard on the temporal and label side of the problem. It has modeled which actions tend to co-occur, split detection queries into instance and boundary branches, and argued about temporal pyramids. All of this runs on inputs that are already spatially collapsed. Work that does keep spatial detail around actions has so far targeted single-label benchmarks or keyframe box prediction. The result is a clean, underexplored question: does giving each label its own view of the frame, at the right spatial scale, help untangle actions that overlap in time?
Core Claim
In dense multi-label temporal action detection, one shared frame vector per time step is a bottleneck. It has three consequences. It caps how well concurrent actions at different spatial scales can be separated. It hurts most on "background" actions like walking or sitting that co-occur with everything. It also hurts on subtle micro-actions. Replacing the shared vector with a class-conditioned readout over a multi-scale spatial token set should raise per-frame and detection mAP. The gains should concentrate on frames with two or more active labels and on classes whose evidence is spatially small. An ablation that keeps the same multi-scale tokens but uses one shared readout should recover little of the gain. If it recovers most of it, the claim is wrong and the benefit is just "more spatial information."
Relation to Prior Work
The current dense multi-label detectors motivate the proposal, because all of them operate on spatially pooled 1D features.
- MS-TCT builds a hierarchical temporal encoder mixing self-attention and 1D convolutions over pooled clip features.
-
Dual DETRs for Multi-Label Temporal Action Detection decodes actions with paired instance-level and boundary-level queries over a 1D sequence of frozen features. - Modeling Multi-Label Action Dependencies (MLAD) attends across classes and across time. It also reports that its dependency modeling fails on ubiquitous background actions.
-
An Effective-Efficient Approach for Dense Multi-Label Action Detection learns co-occurrence structure through a training-only assistant branch. It shows that a non-hierarchical temporal transformer beats a hierarchical one on Charades.
We keep their temporal machinery and change only what enters it. MLAD already builds per-class, per-time representations, but from the same pooled vector. Our per-class representations come from different spatial evidence.
The closest spatial work falls into two groups.
- Single-label temporal localization. PETAL detects the top-K humans with an off-the-shelf detector and merges them into one "group token" per frame. TAL-MTS applies uniform patch self-attention within frames. Both evaluate only on THUMOS-14 and ActivityNet. Both still produce one vector per frame, so neither can give concurrent labels different spatial evidence. That design choice is exactly what we reverse.
- Keyframe localization. AMCRNet shows that multi-scale context pooling helps on AVA. But it predicts actor boxes in keyframes, not start and end times of long overlapping actions.
- Micro-actions. MMAD inserts a fixed-scale spatial depth-wise convolution adapter into the backbone for micro-actions. It still reports that false negatives cluster on short, spatially subtle actions. This motivates scale-aware spatial evidence and gives a strong fine-tuning competitor.
The method's ingredients are borrowed. The scale argument comes from Finding Tiny Faces, which shows that small things are recognized mostly from their surroundings and large things from their internal structure. That is why the readout should be free to choose scale per class. The parallel dilated and multi-kernel branches come from ACNet and AFCN, which use such branches to build multi-scale spatial features cheaply in static images.
The proposal also takes a position on a tension in the literature. MS-TCT and TAL-MTS favor temporal pyramids, while the Effective-Efficient paper finds temporal downsampling harmful. We put the pyramid only in space and keep time at full resolution. This matches both the evidence that small spatial things need multi-scale features and the evidence that dense temporal boundaries need an undownsampled sequence.
One cited number might look contradictory. ProDA reports 71.1 mAP on Charades using scene graphs. That figure appears to be video-level recognition rather than per-frame localization, since the localization numbers on the same dataset sit in the mid-20s. We will not treat it as a localization baseline, and we will state this in the paper.
Research Program
Spatial token set. We keep the backbone's final spatial feature map instead of pooling it. We use VideoMAE or I3D, frozen, as in the prior work. From this map we build a small multi-scale token set per time step. Parallel dilated 3×3 branches run on the map, following ASPP in ACNet. Their outputs are pooled to grids of 1×1, 2×2, 4×4 and 7×7, giving about 70 tokens per time step. Each token carries a scale embedding and a position embedding.
Class-conditioned readout. Each class $c$ owns a learned query $q_c$. Its frame representation at time $t$ is
$$h_{t,c} = \sum_{s,i} \alpha_{t,c,s,i}\, W v_{t,s,i}, \qquad \alpha_{t,c,\cdot} = \mathrm{softmax}_{s,i}\big(q_c^\top K v_{t,s,i} / \sqrt{d}\big).$$
Here $v_{t,s,i}$ is the $i$-th token at scale $s$ and time $t$, $W$ and $K$ are shared value and key projections, $d$ is the key dimension, and $\alpha$ are attention weights over all tokens of all scales. The attention cost grows as classes × time steps × tokens. For Charades (157 classes, 256 steps, 70 tokens) this is a few million dot products per layer, which is small next to the backbone.
Temporal model. The resulting class-by-time grid feeds a non-hierarchical temporal transformer in the style of the Effective-Efficient model, attending along time per class and across classes per time step as in MLAD. The model produces per-frame scores. For detection mAP, the class-time grid instead feeds a DualDETR-style decoder. Each decoder query's class-specific features are gathered from $h_{\cdot,c}$ rather than from one shared sequence.
Analyses. We will relate the learned per-class scale distributions ($\alpha$ summed within each scale) to what the action is about. Charades' object annotations let us split verbs into hand-and-small-object actions and whole-body actions. We will check whether the former put their attention on finer grids and on context around the object, as the tiny-faces argument predicts.
Baselines
Pooled-input reference. The naive reference is the same temporal model fed the standard pooled vector per time step. It uses the same frozen backbone, the same training schedule and the same hyperparameter search budget, with only the input stage changed. This isolates whether any spatial information helps at all.
Strongest per-frame method. The strongest per-frame competitor is the Effective-Efficient model, which reports 26.5% per-frame mAP on Charades and 44.6% per-frame mAP on MultiTHUMOS. We will rerun it on our features rather than only copy numbers, because backbone choice changes results. We will match its training epochs and give it the same number of tuning trials we give our model. Beating it means a gain on Charades per-frame mAP that exceeds the spread across three seeds. It should also be larger on frames with two or more active labels than on single-label frames. MS-TCT (25.4% on Charades, 43.1% on MultiTHUMOS, 33.7% on TSU) serves as the hierarchical-temporal reference.
Strongest detection method. For detection mAP the strongest competitor is DualDETR, which reports 32.6% det-mAP on MultiTHUMOS and 20.8% det-mAP on TSU. We will run it with identical features, number of queries and decoder depth. Our variant changes only where each query's features come from. Beating it means a det-mAP gain on both datasets beyond seed variance.
End-to-end fine-tuning. Because fine-tuning the backbone is a real alternative, we include the DSTA adapter from MMAD on MMA-52. It reports 25.34% average mAP there and a +2.26% gain over AdaTAD on VideoMAE-B. We will give it the same backbone and the same GPU-hours.
Ablations isolating the key ingredient. The shared-readout ablation keeps our exact multi-scale tokens but uses a single query per time step instead of one per class. The single-scale ablation keeps per-class queries but uses only the 7×7 grid without dilated branches. Of everything in this note, these two ablations matter most for the claim.
Why not just…?
Why not just fine-tune the backbone end-to-end, as MMAD and AdaTAD do? This is the most serious threat. A fine-tuned backbone could learn to pack hand-level and body-level cues into different channels of the pooled vector. MMAD's own gain is modest, and it still reports misses concentrated on short, subtle actions. That hints that channel packing does not fully solve it, but it is not proof. We settle this by comparing against DSTA at matched compute on MMA-52. We also stack our readout on top of the fine-tuned backbone. If fine-tuning alone matches us and stacking adds nothing, the bottleneck is not really spatial pooling.
Why not just concatenate spatial-pyramid-pooled features into one vector per frame? This is cheap. It keeps multi-scale information and needs no per-class machinery, and a reviewer might expect it to capture most of the gain. It is essentially our shared-readout ablation, run with both fixed pooling and learned pooling. If it comes within seed variance of the per-class readout, especially on multi-label frames, we will report that the per-class disentanglement claim fails and that the contribution is simply multi-scale spatial input.
Why not just feed object tokens from an off-the-shelf detector, extending PETAL to objects? PETAL shows detector priors help in single-label localization. It notes, however, that detector misses cap performance. AMCRNet reports that pretrained detectors overlook oddly sized or irregular objects. Kitchen utensils and phones on Charades are small and often occluded. We will run a detector-token variant with the same temporal model. Our prediction is that it matches us on classes whose objects the detector covers and falls behind on uncovered ones. That split directly tests whether a learned readout is needed.
Why not just use uniform patch attention within frames, as TAL-MTS does? Uniform patch attention sees spatial detail but at one scale. Finding Tiny Faces argues that small targets need context-heavy templates that differ from those for large targets. The single-scale ablation tests this directly. If it matches the multi-scale version, the scale component should be dropped.
Proposed Experiments
We evaluate on Charades, MultiTHUMOS and TSU for dense overlap, and on MMA-52 for subtle micro-actions. We report per-frame mAP and detection mAP following the metric each baseline used, three seeds each. Three findings are decisive.
First, the per-class readout must beat the shared multi-scale readout. We stratify frames by the number of active labels. The gap should grow with concurrency. If it is flat, the disentanglement explanation is wrong.
Second, the gain should appear on MLAD's background classes such as walking and sitting. MLAD reports that dependency modeling fails there, and spatial scale should help separate whole-body posture from concurrent hand activity. We also look for gains on MMA-52's short, low-coverage micro-actions, where MMAD reports its misses concentrate.
Third, learned scale weights should differ between small-object verbs and whole-body verbs in the direction the tiny-faces argument predicts. If they do not, the model is using spatial detail in some way other than scale selection, and we will say so.
We also report added FLOPs and latency relative to DualDETR's reported 65 ms. This lets readers judge the cost of keeping about 70 spatial tokens per step instead of one.
References
- MS-TCT: Multi-Scale Temporal ConvTransformer for Action Detection
- Dual DETRs for Multi-Label Temporal Action Detection
- Modeling Multi-Label Action Dependencies for Temporal Action Localization
- An Effective-Efficient Approach for Dense Multi-Label Action Detection
- Prior-Enhanced Temporal Action Localization Using Subject-Aware Spatial Attention
- A Multitemporal Scale and Spatial-Temporal Transformer Network for Temporal Action Localization
- Actor-Multi-Scale Context Bidirectional Higher Order Interactive Relation Network for Spatial-Temporal Action Localization
- MMAD: Multi-label Micro-Action Detection in Videos
- Finding Tiny Faces
- Attention Convolutional Binary Neural Tree for Fine-Grained Visual Categorization
- A Novel Attention Fully Convolutional Network Method for Synthetic Aperture Radar Image Segmentation
- Prompt-guided Disentangled Representation for Action Recognition