Paying for New Behaviors, Not New Sessions: Context-Aligned Births in Nonparametric Behavior Clustering

Clustering methods that infer the number of behaviors from data, rather than fixing it in advance, are now standard for segmenting animal motion, driving logs and robot demonstrations. The usual choice is a Dirichlet process, a prior that lets any new observation join an existing cluster or open a fresh one. Their selling point is open-ended discovery: a new movement pattern gets its own cluster without disturbing the old ones. That same property becomes a liability when data arrive from many recording sessions, animals, drivers or maps. A session with slightly different camera placement, or a driver who simply keeps a longer following distance, looks "new" to the model. The model then opens clusters that track who or where the data came from, not what the agent was doing. Fixed-cluster models can only mislabel nuisance variation. Unbounded models can also grow without limit because of it, so the count of discovered behaviors rises with the number of sessions rather than the number of behaviors.

We propose a behavior segmenter that charges for a new cluster only when the data cannot be explained by an existing behavior after a small, group-specific alignment. Concretely, it is a hierarchical Dirichlet process hidden semi-Markov model. In plain terms, this is a segmenter that cuts a time series into an unbounded set of recurring states, shares those states across recordings, and models how long each state lasts. We add to it a low-dimensional transform per recording group, with a strong prior pulling it toward the identity. The claim is testable. Under this model, the number of discovered behaviors should flatten as sessions are added, and labels should carry little session information. A genuinely new behavior that appears in a new session should still be born as its own cluster rather than absorbed into the alignment.

Core Claim

Existing work splits the problem in two. Learning Disentangled Behavior Embeddings (VDBE) uses two encoders, one for static context and one for pose, to separate animal identity and session from motion. It then fits a finite Gaussian mixture with a hand-set state count. It reports motif accuracy of about 79.5% on a mouse hand-reach task, versus about 53% for BehaveNet. It also reports that session identity, perfectly decodable from naive embeddings, drops to about 28% decodability. This is the strongest evidence that confounding is the central failure. However, the motif count is fixed and the context is assumed constant within a clip. A Bayesian Nonparametric Framework For Learning Disentangled Representations (Bayes-QLAE) places Dirichlet process priors on latent factors and discovers how many modes each factor has. It works on static images only, with no dynamics.

The driving literature has the opposite profile. Nonparametric segmenters such as the sticky HDP-HMM in A Tempt to Unify Heterogeneous Driving Databases using Traffic Primitives and Multi-Vehicle Interaction Scenarios Generation with Interpretable Traffic Primitives and Gaussian Process Regression fit raw kinematics with no notion of who produced them. Driving Style Analysis Using Primitive Driving Patterns With Bayesian Nonparametric Approaches shows that explicit duration modeling (HDP-HSMM) fits car-following data better than the sticky HDP-HMM. It then falls back on K-means and hand thresholds to make primitives comparable across 18 drivers. That fallback signals that the raw nonparametric primitives did not line up across people.

Our claim is that the birth decision itself must be made in context-aligned coordinates. Disentangling first and clustering second, as VDBE does, is not enough, and neither is clustering raw signals, as the driving papers do. The multi-agent works Fast Teammate Adaptation in the Presence of Sudden Policy Change and Collaboration with Dynamic Open Ad Hoc Team via Team State Modelling run Chinese-restaurant-process clustering over raw state-action traces. The file notes that this leaves them open to map and target-location confounds. Their failure is the same one, and we treat them as motivation rather than as a testbed.

Model

Let $x_t^g$ be the feature vector at time $t$ in recording group $g$, which may be a session, an animal or a driver. Let $z_t$ be its behavior state, drawn from an HDP-HSMM whose states and transition structure are shared across groups. Each state $k$ has a duration distribution and a canonical emission mean $\mu_k$ and covariance $\Sigma_k$. Each group has an affine transform $(A_g, b_g)$ that is shared across all states, so the observation model is

$$ x_t^g \sim \mathcal{N}\big(A_g \mu_{z_t} + b_g,\; A_g \Sigma_{z_t} A_g^\top + \Psi\big), $$

where $\Psi$ is diagonal sensor noise. $A_g$ is restricted to diagonal-plus-low-rank and given a prior centered at the identity, with $b_g$ centered at zero. Because the transform is shared by every state in a group, it can absorb a global shift, such as a driver's preferred headway or a camera's scale. It cannot absorb one new pattern while leaving the others fixed. That asymmetry is what lets a real new behavior still trigger a birth.

Group-specific duration scaling handles tempo differences on the time axis. This is a lightweight cousin of the continuous warping in Bayesian Nonparametric Dynamical Clustering of Time Series. That work clusters one-dimensional ECG with per-segment Gaussian-process time warping. We borrow its idea that alignment should come before cluster assignment, but replace per-segment warps with one rate parameter per group and state. This keeps inference cheap in many dimensions.

For video, $x_t^g$ is the output of a pose encoder trained jointly with this prior through a variational objective. The prior's assignment and alignment errors feed back into the encoder. For low-dimensional kinematics such as car-following range, range rate and acceleration, the model runs directly on the features. Inference uses blocked Gibbs sampling with a weak-limit truncation for the kinematic case and structured variational inference for the video case. We deliberately do not use the small-variance hard-assignment limit of Small-variance asymptotics for non-parametric online robot learning. The quantity we want to measure is the posterior probability that a segment belongs to a new state versus an aligned old one, and hard assignments discard exactly that. The hard-assignment variant is a natural fast follow-up once the birth behavior is understood.

Research Program

The work has three parts. The first is a controlled synthetic study. Known motion primitives are rendered through per-group affine and tempo distortions of graded strength. In some held-out groups we also inject one genuinely new primitive. This tells us when alignment absorbs real novelty, and the main risk of the design is that a flexible $A_g$ explains away a new behavior.

The second part is the mouse video setting from VDBE, with the hand-reach and wide-field calcium imaging datasets. Here nuisance variation is visual and nonlinear and the encoder is learned. The third part is naturalistic car-following from the Safety Pilot Model Deployment data, the 18-driver subset used by Wang et al. Here the group is the driver, and the question is whether shared primitives emerge without the K-means post-processing that paper needed. In all three, we track a curve that prior papers do not report: the number of occupied states as a function of the number of groups seen, along with held-out-group birth rates.

Baselines

The naive reference is a sticky HDP-HMM and an HDP-HSMM fit to pooled features with no group information. This is the configuration used by Zhu et al. and Wang et al. It is run with the same truncation level, sampler iterations and concentration priors as our model, on the same features. We expect it to show state counts that rise with the number of groups. Any method that does not flatten that curve has not addressed the problem.

The strongest existing method on the video data is VDBE, whose reported hand-reach motif clustering accuracy is about 79.5%, with BehaveNet at about 53% and session decodability of about 28%. We rerun VDBE on the same splits and give it a fair chance at choosing its state count: we sweep k over a grid and select by held-out likelihood, with the same total GPU budget as our joint model. Beating it means matching or exceeding its motif accuracy at an inferred rather than tuned state count, with equal or lower session decodability from the state labels. It also means fewer spurious births on held-out animals. For SPMD there is no reported numeric result on Lacuna for primitive accuracy or log-likelihood. Wang et al. report only that the HDP-HSMM beats the sticky HDP-HMM in likelihood. So we reproduce their HDP-HSMM plus K-means pipeline and compare held-out-driver predictive log-likelihood and cross-driver primitive consistency directly.

Two ablations isolate the key ingredient. The first is a two-stage pipeline: VDBE pose embeddings, frozen, followed by an HDP-HSMM. This has nonparametric births but no alignment during the birth decision and no feedback to the encoder. The second is our full model with $A_g$ fixed to the identity, which leaves the duration scaling and joint training in place but removes the alignment. If the full model does not beat both ablations on held-out-group births, the alignment-in-the-birth-decision claim is wrong.

Why not just…?

Why not just z-score features per session or driver, then run an HDP-HSMM? For low-dimensional kinematics this is a serious threat. Per-group standardization is a special case of our transform with $A_g$ diagonal and fixed by moment matching. It may well match us on SPMD. It has two weaknesses. Its statistics depend on that group's behavior mix, so a cautious driver who rarely brakes hard gets their hard braking rescaled toward everyone else's moderate braking. Our model instead estimates the transform jointly with the state assignments. Standardization also cannot address nonlinear visual nuisance in video. We include it as a baseline in every setting. The synthetic study varies behavior-mix imbalance across groups precisely to test whether the imbalance failure appears. If it does not, we will report that standardization suffices for kinematics.

Why not just take VDBE and choose k by held-out likelihood? VDBE already removes much session information, and a tuned k could match our accuracy on a fixed dataset. What a tuned fixed k cannot do is respond to a new animal or session that brings a new behavior without retraining. SOSC's online growth from one to six states makes this point in robot demonstrations. Our held-out-group injection experiment tests it directly. If tuned-k VDBE matches us on both static accuracy and novelty detection, the nonparametric component is unnecessary here and we will say so.

Why not just disentangle first and cluster the frozen embeddings with a nonparametric model? This is the most natural combination of the two directions and the closest competitor. Its weakness is that VDBE's context encoder assumes context is constant within a clip. Any residual nuisance in the pose embedding then becomes a cheap reason to open a new state. The two-stage ablation is run with identical encoders and compute, and the decisive comparison is births per held-out group. If the two-stage pipeline flattens the cluster-count curve as well as our model, the contribution reduces to duration modeling, and the paper becomes a negative result about joint training.

Proposed Experiments and Decisive Checks

On synthetic data, the first check is that the number of occupied states stays near the true count as groups increase from 2 to 50, while the naive HDP-HSMM grows. The second check is that injected novel primitives are born with recall comparable to the no-nuisance setting across distortion strengths. We will report the distortion level at which alignment starts absorbing novelty. That level is the model's honest operating envelope.

On hand-reach and calcium imaging video, we report motif accuracy against the provided annotations, using the same matching protocol as VDBE. We also report session and animal decodability from state labels, inferred state count, and births on held-out animals under leave-one-animal-out evaluation.

On SPMD, we evaluate in a leave-one-driver-out setting. The metrics are held-out-driver predictive log-likelihood, the fraction of held-out driving explained by existing primitives, and divergence between drivers' primitive-usage distributions computed on shared primitives. Wang et al. computed that divergence only after K-means relabeling. We also inspect whether the learned per-driver offsets correlate with simple style statistics such as mean time headway. If they do, the transform captures style rather than behavior, as intended.

The proposal is falsified under two conditions. The first is if per-group standardization or the two-stage pipeline matches the full model on held-out-group birth rates in both real domains. The second is if the full model's suppression of spurious births comes at the cost of missing injected novel behaviors.