Models that recognize human actions or search for people by text work better when they are told where the body is. Current systems get that information from a pose estimator, a separate network that finds joints such as wrists, knees and the head. That dependency is costly and fragile. The estimator has to run at test time, and it fails in exactly the frames that matter most: cluttered rooms, occluded limbs, poor light, crowds. Synthetic human data is now common, but almost all of it is used only as extra images. Yet when a person is rendered from an animated 3D avatar, the renderer already knows which pixel belongs to which body part. It knows this exactly, with no estimator involved and no extra cost.
This note proposes to use that knowledge as a training signal. During training on synthetic data, the recognition model's internal attention is pushed toward the rendered body-part masks. On real data, and at test time, no pose is used at all. The claim is that a plain video transformer taught this way matches pose-at-test-time models on ordinary footage and beats them on the frames where pose estimators break. The experiments are designed so that an estimator-based version of the same idea can falsify this claim.
Core Claim
Two lines of work sit side by side without informing each other.
Pose-guided attention injects keypoints into recognition networks. Learning Pose-aware Representations in Vision Transformers for Understanding Activities of Daily Living does this with its Pose-aware Attention Block (PAAB), which restricts late-layer self-attention to image patches containing OpenPose keypoints. Toyota Smarthome does it with a skeleton-guided spatial attention branch. Beyond Walking does it with a pose-aware image encoder for anomaly search. All three need pose at inference, so they inherit the estimator's failures.
Part-Aware Bottom-Up Group Reasoning takes a different route. It removes test-time pose by using estimator outputs only as training-time supervision for part attention. This is our closest methodological ancestor. The difference is that its training masks are themselves estimator outputs, which are noisy precisely where occlusion is heavy.
The synthetic-data line has the opposite gap. Synthetic Human Action Video Data Generation with Pose Transfer renders real motions onto 3D Gaussian avatars and trains generic ResNet and SlowFast models. The pedestrian corpora MALS and SDA use pose only as a filter that throws away malformed images. None of these papers treats the generator's knowledge of body geometry as supervision for the downstream model's attention.
Our specific claim follows from these two gaps. Part masks that are exact by construction, taken from avatar rendering, are a better attention teacher than estimator pseudo-masks. The benefit should concentrate on real test frames where estimator confidence is low. If the gain appears only as general accuracy, and equally with estimator masks, then the "exact labels" hypothesis is wrong and what remains is a data-volume effect.
Research Program
We build one training objective and one data pipeline, then test them against the alternatives that could explain away any gain.
The objective. The backbone is a TimeSformer-style video transformer, chosen so it matches the PAAB setting. Let $A_{l,h}$ be the attention distribution from the class token over spatial patches, in layer $l$ and head $h$. Let $M_{p}$ be the rendered mask for body part $p$ (head, torso, left and right arm, legs), downsampled to the patch grid and normalized to sum to one. Let $p(h)$ be a fixed assignment of heads to parts. For synthetic clips we minimize
$$\mathcal{L} = \mathcal{L}_{\text{task}} + \lambda \sum_{l \in \mathcal{S}} \sum_{h} \mathrm{KL}\left(M_{p(h)} \,\|\, A_{l,h}\right),$$
where $\mathcal{L}_{\text{task}}$ is the classification or contrastive loss, $\lambda$ weights the attention term, and $\mathcal{S}$ is the set of supervised layers.
Two design choices follow from prior evidence. We restrict $\mathcal{S}$ to late layers, following PAAB's finding that pose constraints help after layer 12 and hurt when imposed on low-level layers. We leave half the heads unsupervised, so the model can still attend to context. On real clips only $\mathcal{L}_{\text{task}}$ applies, and inference uses no pose.
Handling objects the person uses. Masks that cover only the body would suppress the cups and kettles that define many daily activities. PAAB's own conclusion flags this blind spot, and the +21.4% "Pour from kettle" gain in Toyota Smarthome shows how much such actions depend on hand regions. We therefore dilate the hand-part masks by a reach radius. We treat the radius as a studied variable, not a fixed constant.
The data. The primary source is the avatar-rendered RANDOM People data from Synthetic Human Action Video Data Generation with Pose Transfer, re-rendered with per-part segmentation channels. Rendering from the avatar mesh makes these masks exact. That paper's pipeline is single-person and has no objects, so we scope the main study to single-person daily activities rather than overclaim.
A secondary source tests whether the idea survives when labels are not exact. Pose-conditioned diffusion does not reliably follow its conditioning pose: Follow-Your-MultiPose reports only 35.32% pose accuracy. For diffusion images we therefore keep a mask only when the conditioning skeleton and an independent estimator run on the output agree. We call these agreement-filtered masks.
Baselines
Naive reference points. The weakest point is the plain TimeSformer trained on real Toyota Smarthome clips alone. The more important naive reference is the same TimeSformer trained on real plus RANDOM People synthetic clips, with $\lambda = 0$. This is the "just add synthetic data" model, and it is essentially the regime in which the pose-transfer paper reports ResNet accuracy rising from 20% to 51% on Toyota Smarthome. Beating it isolates the attention loss from the data itself.
Strongest pose-at-test-time method. This is PAAB (Learning Pose-aware Representations in Vision Transformers). It reports a 3.0-point mean class accuracy gain over TimeSformer on Toyota Smarthome; no reported absolute mCA was found on Lacuna for PAAB. The strongest reported absolute number in the file on the cross-subject protocol is 54.2% accuracy for skeleton-guided separable attention on I3D (Toyota Smarthome).
We rerun PAAB with the same backbone and the same real-plus-synthetic data, using OpenPose keypoints at test time. It gets an equal hyperparameter search budget, which we set at 20 trials per method over learning rate and the method-specific weight. Every method uses the same number of training steps. Our success criterion here is to match PAAB overall, within one standard deviation across three seeds, while using no pose at inference.
Strongest pose-free competitor. This is the privileged-supervision recipe of Part-Aware Bottom-Up Group Reasoning, transplanted to our backbone. The loss is identical, but the masks come from an estimator run on the synthetic and real frames. This is the ablation that isolates our key ingredient, mask exactness. We expect to beat it by a clear margin on the low-confidence subset described below, and to tie it elsewhere.
Controls. Two further controls check whether the anatomical shape of the target matters. One replaces part masks with filled person bounding boxes. The other uses masks shuffled across clips, which keep the mask statistics but carry no information about the actual body.
Why not just…?
Why not just scale synthetic data and train a plain transformer? Contrastive Transformer Learning with Proximity Data Generation argues that a plain dual transformer with synthetic hard negatives reaches strong person-search results without any local alignment. Beyond Walking reports the opposite for behavior-centric retrieval, where removing its pose encoder hurts. The two findings may differ because appearance matching is global while action is local. Our $\lambda = 0$ baseline, trained on identical data, settles this directly for action recognition. If it ties our model everywhere, including on occluded frames, the proposal fails.
Why not just supervise attention with estimator pseudo-masks, as the part-aware social-interaction work does? This is the real threat. Estimator masks are cheap and apply to real data too, so if they are good enough, exact rendered masks add nothing. Our prediction is narrow. Estimator masks should be adequate on clean frames and harmful where estimator confidence is low. The stratified evaluation below tests this, and we will report a null result there as a null.
Why not just keep the pose estimator at test time, as PAAB and Beyond Walking do? On clean benchmarks it may well be as accurate as our model. The argument for our approach rests on two things: inference cost, and robustness when keypoints go missing or get swapped. We measure both, by comparing throughput and by comparing accuracy on the low-confidence subset. If PAAB wins on that subset, test-time pose is simply the better choice, and we will say so.
Proposed Experiments
Main evaluation. The main benchmark is Toyota Smarthome, cross-subject and cross-view, reporting mean class accuracy. NTU RGB+D, cross-subject, serves as a check on cleaner lab footage, including the one-shot protocol where the pose-transfer paper reports 7% rising to 14%.
The decisive check. We split each real test set by OpenPose mean keypoint confidence and report the lowest tercile separately. This low-confidence tercile carries the paper's central claim. The model should beat estimator-mask supervision and PAAB there, while staying within noise of them on the top tercile. The same pattern should hold on synthetic clips with rendered occluders, where the true masks are known.
Checking that attention actually moved. We measure attention-mass-inside-body on held-out renders. On real frames we compare against sparse manually annotated part masks for 500 frames, so that a gain cannot be attributed to the loss while the attention itself stays unchanged.
Hand dilation and objects. We sweep the hand dilation radius and report object-centric classes such as "Pour from kettle" separately. This tests whether body-only supervision costs accuracy on activities defined by an object, and whether dilation recovers it.
Secondary study: text-based anomaly search. We regenerate a subset of the PAB training data with pose-conditioned diffusion and agreement-filtered masks. We then train a pose-free retriever and evaluate on the real PAB test set and the UCC out-of-distribution set. The comparison point is the reported CMP results in Beyond Walking: 84.93% recall@1 on PAB and 54.12% on UCC, both obtained with pose at test time. A pose-free model that approaches these numbers, especially on UCC, would show the idea extends beyond exact avatar masks to filtered diffusion masks.
References
- Learning Pose-aware Representations in Vision Transformers for Understanding Activities of Daily Living
- Toyota Smarthome: Real-World Activities of Daily Living
- Beyond Walking: A Large-Scale Image-Text Benchmark for Text-based Person Anomaly Search
- Part-Aware Bottom-Up Group Reasoning for Fine-Grained Social Interaction Detection
- Synthetic Human Action Video Data Generation with Pose Transfer
- Towards Unified Text-based Person Retrieval: A Large-scale Multi-Attribute and Language Search Benchmark
- Minimizing the pretraining gap: Domain-aligned text-based person retrieval
- Follow-Your-MultiPose: Tuning-Free Multi-Character Text-to-Video Generation via Pose Guidance
- Contrastive Transformer Learning With Proximity Data Generation for Text-Based Person Search