Ecologists increasingly want to measure behavioral plasticity, meaning how much an animal changes what it does, how often, and for how long when its environment changes. Grooming harder under parasites and dropping costly defenses where enemies are absent are two examples. Automated video classifiers now make it possible to score behavior over thousands of hours instead of the few hours a human can watch. The catch is that comparisons across sites, labs, or elevations are exactly where these classifiers break: a model trained at one place makes different mistakes at another. When those mistakes differ by site, the measured "difference in behavior" is partly a difference in the measuring instrument.
This note proposes treating an automated behavior classifier as a measurement device with its own site-specific error. That error is estimated from a small number of human-scored clips per site. Each site's true bout rates and bout durations are then recovered by inverting the error model, rather than by trusting the classifier's raw output. The central worry is concrete. A classifier that flickers on and off at a new site splits one long grooming bout into several short ones. That produces more bouts of shorter duration, which is the same signature Zhang et al. used as evidence of metabolic exhaustion in mite-stressed termites. Domain shift can therefore manufacture a biological finding, and current pipelines have no way to detect this.
Core Claim
Bout-level behavioral statistics (how often bouts start, how long they last, which behavior follows which) are far more sensitive to cross-site classifier error than frame-level accuracy suggests. A modest per-site audit, combined with a temporal error model, can recover unbiased cross-site plasticity estimates with honest uncertainty, and better classifiers alone cannot do this.
The nearest existing work stops short of this question. Benchmarks of cross-location shift, WILDS and iWildCam 2021, show that models degrade at unseen camera sites, but they score species identity and counts from bursts of still images. LoTE-Animal spans an elevation gradient with action labels but treats actions as fixed benchmark targets. None of these asks whether a downstream behavioral estimate is biased. The goal here is not a more accurate classifier; it is a correct comparison between sites given whatever classifier one has.
Research Program
The pipeline has three parts.
The classifier. Any frame-level behavior classifier can serve: the frozen VideoPrism model with a linear head from Sun et al., or a 3D convolutional network in the style of Geuther et al.. It produces per-frame behavior probabilities at every site.
The per-site audit. At each site, humans score a small set of short windows. A shared calibration set of clips is scored by every auditor. This guards against the problem documented by Tjandrasuwita et al., whose synthesized programs showed that individual experts apply systematically different temporal rules. Without the shared set, an auditor's style could pose as a site effect.
The error model. The model is a hidden semi-Markov model, meaning a model in which the animal's true behavior is a hidden sequence of bouts, each bout drawn with an explicit duration distribution. The classifier's outputs are treated as noisy readings of that hidden sequence. The parameters of the true behavior are shared in form but estimated per site: bout-start rates $\lambda_s$, bout-duration distributions $D_s$, and transition probabilities. The reading model is per-site, and it includes both confusion between behaviors and the classifier's tendency to flicker or lag at bout edges.
Audit windows pin down the reading model at each site. The unlabeled footage then constrains the true-behavior parameters through expectation–maximization. Plasticity is reported as the difference between sites in true-process parameters, for example
$$\Delta_{AB} = \log \frac{\mathbb{E}[D_A]}{\mathbb{E}[D_B]},$$
where $\mathbb{E}[D_s]$ is the estimated mean true bout duration at site $s$. Uncertainty is carried through from the audit by bootstrapping over audit windows.
Choosing audit windows. A second contribution is a rule for picking which windows to audit. The rule targets the windows where uncertainty about the reading model most inflates the variance of $\Delta$. In practice these are bout boundaries and low-confidence stretches, rather than random clips. This matters because, as the iWildCam 2021 paper notes, most field footage is empty, so randomly sampled audits waste most of the human time.
Baselines
Raw classifier statistics. The naive reference point computes bout rates and durations directly from thresholded classifier output, which is what current practice does. It uses the same classifier and footage as the proposal. Any method that cannot beat it on bias is useless.
The strongest scoring model. The strongest available classifier is frozen VideoPrism with a trained head, which Sun et al. report at 71.2 mAP on the Calico mouse dataset against 52.6 mAP for a specialized baseline. The 3D network of Geuther et al. reports 93.7% accuracy and 0.984 AUC within a single institution. For cross-site bout statistics, no reported result was found on Lacuna for either model. Both will be run as plug-in estimators and as the classifier inside our correction.
Per-site fine-tuning. The most important competitor fine-tunes the classifier at each site using the same audit labels and the same compute and tuning budget, then computes statistics naively. It spends human effort on improving the model instead of on correcting the estimate.
Temporal smoothing. This baseline applies median filtering or a plain hidden Markov model, following the transition-modeling approach of Jiang et al., with no site-specific reading model.
Confusion-matrix correction. This baseline adjusts each site's time budget using that site's estimated confusion matrix, but has no bout structure.
Audit-only estimate. This baseline computes statistics from the human-scored windows alone.
Ablations. Two ablations isolate the key ingredients. One removes the edge-flicker component of the reading model. The other replaces targeted audit selection with random selection.
What counts as beating them. The proposal wins if it achieves lower bias and properly covering 90% intervals for $\Delta$ under the null and planted-shift tests below, at equal human annotation cost.
Why not just…?
Why not just hand-score a subsample at each site? This is unbiased if auditors are consistent. However, bout durations are long-tailed, and each audit window truncates bouts at its edges, so durations estimated from short clips are both noisy and biased toward shorter values. The audit-only estimator is a baseline. If it matches our variance at equal cost, the proposal fails, and we will report that.
Why not just build a better or fine-tuned classifier? Better accuracy shrinks the bias but does not remove it, and nothing in such a pipeline reports the bias that remains. WILDS found that methods designed for robustness to distribution shift, such as invariant risk minimization (IRM) and CORAL, failed to beat ordinary training on 8 of 10 real-world shifts. Animal Kingdom reports only 39.7% action accuracy on unseen animals for its best method. Per-site fine-tuning is a real threat. The fine-tuned arm at equal label budget will decide whether this threat holds.
Why not just smooth the predictions? Smoothing removes flicker at a fixed scale that is not specific to any site. That scale either merges true short bouts or misses longer flicker, and it does so differently at each site. The hidden Markov model baseline tests this directly.
Why not standard confusion-matrix correction? This alternative is sound for time budgets, and we expect it to tie with our method there. It cannot correct bout counts or durations, because those depend on how errors are arranged in time, not only on how often they occur.
Proposed Experiments
The null test. Behavior is held fixed while the "site" changes, so any measured plasticity is artifact. We take the Calico mouse videos and the open-field videos from Geuther et al. and render each clip under several site-like perturbations: background replacement, lighting, viewpoint crops, and compression. The true $\Delta$ is zero by construction. We measure how often each method declares a significant shift. We predict that raw classifier output will declare a spurious "more, shorter bouts" shift, and that our corrected estimates will keep false declarations near the nominal rate.
The planted-shift test. Bout sequences are resampled to create a known change in duration, and footage is stitched to match. Each method's recovery of the known $\Delta$ is scored by bias and interval coverage.
The auditor-style test. We use the CalMS21 Task 2 annotator-style data, where the published fine-tuned model reports F1 of 0.813 ± 0.003. Different annotators are assigned to different pseudo-sites. We test whether the shared calibration set prevents annotator style from being read as plasticity.
A real gradient. We apply the pipeline to LoTE-Animal action clips across its 1,150–6,250 m elevation range, where wild pose estimation reaches only 43.74% AP. The question is how much of any apparent elevation trend in action budgets survives correction.
The decisive check. The proposal stands or falls on whether, at equal human effort, it beats per-site fine-tuning on interval coverage in the null and planted tests. If it does, the pipeline is ready for a multi-colony termite or ant study of the kind Zhang et al. and Tragust et al. scored by hand.
References
- The mite Acarus farris inducing defensive behaviors and reducing fitness of termite Coptotermes formosanus: implications for phoresy as a precursor to parasitism
- WILDS: A Benchmark of in-the-Wild Distribution Shifts
- The iWildCam 2021 Competition Dataset
- LoTE-Animal: A Long Time-span Dataset for Endangered Animal Behavior Understanding
- Video foundation models for animal behavior analysis
- Action detection using a neural network elucidates the genetics of mouse grooming behavior
- Interpreting Expert Annotation Differences in Animal Behavior
- Context-Aware Mouse Behavior Recognition Using Hidden Markov Models
- Animal Kingdom: A Large and Diverse Dataset for Animal Behavior Understanding
- The Multi-Agent Behavior Dataset: Mouse Dyadic Social Interactions
- Pupal cocoons affect sanitary brood care and limit fungal infections in ant colonies