Scoring animal behavior from video still depends heavily on people. Human observers disagree with one another, tire over long sessions, and cannot keep pace with genetic screens or around-the-clock farm monitoring. Automated scorers usually take one of two routes. Some read the animal's skeleton, meaning tracked body keypoints such as snout, paws, and tail base. Others read raw video through large pretrained networks. Both routes lose accuracy when the lab, camera, annotator, or species changes. The obvious fix is to combine the two, but combining them naively tends to fail in a specific way. Video features are high-dimensional and already contain most of what the skeleton says, together with cage, lighting, and background detail that differs from one setting to the next. A fused model can then rely on video for everything, a failure called modality dominance. It ignores the skeleton and learns to recognize the setting instead of the behavior.
Our proposal is to remove that overlap before fusing. We first subtract from the frozen video features the part that the skeleton can predict. From what remains, we keep only the directions that predict behavior consistently across the training domains and that do not reveal which domain a clip came from. The skeleton then reads this small residual through gated cross-attention, an attention layer in which the pose stream asks the questions, the video residual supplies the answers, and a learned gate sets how much of each answer is used. The resulting claim can be tested. Cross-domain failure in pose-plus-video behavior models comes from redundant video dimensions, so removing them should improve transfer to unseen species and annotators more than generic multimodal regularization does, and without the target-domain validation that such regularization often requires.
Core Claim and How It Differs from Prior Work
Three groups of earlier papers each cover part of this problem.
The first group motivates the gap. Animal Kingdom shows how severe cross-species failure is. Its CARe model separates features shared across species from species-specific ones and reaches only 39.7% action accuracy on unseen species, and pose accuracy falls from 66% to 39.3% PCK on unseen sub-classes. CARe works on RGB video alone. Video Foundation Models for Animal Behavior Analysis shows that frozen VideoPrism features with a linear probe beat specialized pipelines (71.2 versus 52.6 mAP on Calico), but it applies no pose fusion and no compression to those features. CalMS21 supplies an annotator-shift task. It also notes that pose-only models miss contextual cues such as bedding and cage fixtures, which is exactly the information video should add.
The second group supplies the method. Gated cross-attention for drug-target interaction combines a learned gate with sparse attention so that a large protein representation does not swamp a small drug representation. We reuse that gate, with pose in the role of the small stream. Greedy eigencomponent selection shows that in deep features the highest-variance directions often encode nuisance variation, and that a handful of directions chosen for the task beat PCA. We adopt the idea of task-driven component selection. However, that method selects components using labeled target AUROC, and we select only with signals from the training domains.
The third group constrains the design. Cross-modal Representation Flattening (CMRF) identifies modality competition as the core obstacle to multimodal domain generalization. It addresses competition by mixing representations across modalities and distilling from a teacher that tracks a running average of the model's weights, reaching 71.91% on Human-Animal-Cartoon against 63.11% for joint training and 64.72% for Sharpness-Aware Minimization (SAM). It treats all representation dimensions alike and tunes modality weights on a validation set. CIBR removes modality-specific noise with a penalty based on a neural estimate of mutual information. That estimator is trained adversarially against the model, and the paper notes it is unstable, so we avoid it.
Our approach departs from these in one respect. We treat the overlap between modalities as an explicit, measurable object and remove it in closed form. We do not regularize the loss landscape, and we do not estimate mutual information.
Research Program
Let $v_t \in \mathbb{R}^D$ be frozen video-foundation features for time window $t$, and let $p_t$ be features computed from the skeleton trajectory over the same window. We fit a ridge regression $W$ from pose to video on the training domains and keep the residual
$$r_t = v_t - W p_t .$$
The residual holds what video carries beyond pose: context, the partner animal's appearance, objects, and nuisance variation. We eigendecompose the residual covariance and score each component $j$ by
$$s_j = \min_{d} \mathrm{F1}_d(j) - \lambda \, \mathrm{Acc}_{\mathrm{dom}}(j),$$
where $\mathrm{F1}_d(j)$ is held-out behavior F1 on training domain $d$ when component $j$ is added to the current pose-plus-selected-components probe, and $\mathrm{Acc}_{\mathrm{dom}}(j)$ is how well that component alone predicts domain identity. Taking the worst-domain F1 rewards components that are useful everywhere and not only in one setting. We add components greedily until leave-one-training-domain-out F1 stops rising, which keeps every choice inside the training data. The fusion head then uses pose tokens as queries over the selected residual tokens, with the output combined as $g \odot \text{context} + (1-g)\odot \text{pose}$ following the gated cross-attention design. The per-frame gate $g$ gives a readable record of when the model consults video.
The analysis side asks three questions. Does dominance actually occur in naive pose-plus-video fusion on these datasets? We test this by probing the fused model's internal representation separately for pose-derived and video-derived accuracy. How much of the video feature is pose-predictable? And how much domain identity can be decoded from the raw video features compared with the residual and the selected subspace?
Baselines
The naive reference is the skeleton-only temporal convolutional network from CalMS21. Its reported Task 1 F1 is 0.793 ± 0.011, and the best Task 2 (annotator transfer) model reaches 0.813 ± 0.003. Any fusion method has to clear this, because it shows what pose alone provides.
The strongest video-only competitor is frozen video-foundation features with a linear probe, following Video Foundation Models for Animal Behavior Analysis. The only reported number is 71.2 mAP on Calico. We found no result on Lacuna for this probe on CalMS21 or the Animal Kingdom unseen-species split, so we will run it ourselves with the same backbone our method uses.
The strongest existing domain-generalization method for animal action recognition is CARe, at 39.7% unseen-species accuracy on Animal Kingdom, ahead of Mixup at 36.2% and Episodic-DG at 34.0%. Beating it means exceeding 39.7% on the same split. We will also report our own rerun of CARe so that differences between their setup and ours do not decide the comparison.
The strongest generic multimodal domain-generalization method is CMRF, applied to pose plus video. On Human-Animal-Cartoon we compare against its reported 71.91%.
Fairness rules apply to every comparison. All methods use the same frozen video backbone and the same pose estimator. On Animal Kingdom that estimator is trained only on seen species, so unseen-species pose is noisy, as it would be in deployment. Every method gets the same number of hyperparameter trials, selected by leave-one-training-domain-out validation. No method sees a target-domain validation set, and that includes CMRF's modality weights.
Ablations isolate each ingredient. One variant drops the pose residualization and uses raw $v_t$. Another drops the domain-predictability term by setting $\lambda=0$. A third replaces greedy selection with top-k PCA at equal $k$. A fourth replaces gated cross-attention with concatenation of pose and selected residual. Our method beats a baseline only if it improves worst-domain F1 or accuracy by more than the seed-to-seed standard deviation over five seeds.
Why not just…?
Why not just concatenate pose and video features? This is the main threat to the attention component. The concatenation ablation applied to the selected residual tests it directly. CMRF's gap of 63.11% versus 71.91% suggests that naive joint training is weak when inputs overlap. Our hypothesis, however, is that most of the gain comes from residualization, which acts before fusion. If concatenating residuals matches gated cross-attention, we will report that the contribution lies in redundancy removal and drop the attention component.
Why not just use a frozen video foundation model with a probe? It is strong in-domain. The file has no evidence on how it handles unseen species or annotator shift, and CIBR's gains from compressing CLIP features suggest that uncompressed foundation features transfer suboptimally. The domain-decodability analysis will show whether raw video features encode domain identity, and the unseen-species split will show whether that costs accuracy.
Why not just apply PCA to the video features? The eigencomponent-selection result shows that high-variance deep-feature directions often carry nuisance variation. Our PCA-at-equal-$k$ ablation tests whether that holds here. If PCA matches greedy selection, the selection criterion is unnecessary.
Why not just crop the video around the animal using the skeleton? This is a real competitor, because cropping removes background without any learning. It also discards context that CalMS21 identifies as missing from pose, such as cage fixtures and the second animal's position, and on unseen species it inherits the pose estimator's errors. We will run pose-box cropping with the same probe. If it matches our method, the residual is doing nothing beyond what cropping does.
Why not just use CMRF or CIBR? CMRF needs target-validation weighting and does not touch overlapping dimensions. CIBR's mutual-information estimator is unstable. Our method runs both as baselines under the no-target-validation rule.
Proposed Experiments and Decisive Checks
The main experiments use the Animal Kingdom unseen-species split, CalMS21 Tasks 2 and 3, and Human-Animal-Cartoon with audio in place of pose, which tests whether the residualization principle extends to other modality pairs.
Three results would decide the outcome. First, naive fusion must show measurable dominance: a pose probe on the fused representation should fall below the pose-only model. If it does not, the motivation fails. Second, domain decodability should drop sharply from $v_t$ to the selected residual while behavior F1 holds steady. Third, on unseen species the method should exceed both CARe's 39.7% and the cropping baseline, with the gain disappearing when residualization is removed.
A qualitative check supports these. The gate should open more for behaviors that look alike from posture but differ in context, such as eating, drinking, and grooming, than for locomotion. If any of the three decisive checks fails, we will report that the redundancy account of cross-domain failure is wrong for these data.
References
- Animal Kingdom: A Large and Diverse Dataset for Animal Behavior Understanding
- Video foundation models for animal behavior analysis
- The Multi-Agent Behavior Dataset: Mouse Dyadic Social Interactions
- An Interpretable Framework for Drug-Target Interaction with Gated Cross Attention
- Gaussian Image Anomaly Detection with Greedy Eigencomponent Selection
- Cross-modal Representation Flattening for Multi-modal Domain Generalization
- CIBR: Cross-modal Information Bottleneck Regularization for Robust CLIP Generalization