Distinguishing discrete and continuous behavioral variability using warped autoregressive HMMs

Julia Costacurta, Lea Duncker, Blue Sheffer, Winthrop Gillis, Caleb Weinreb, Jeffrey E. Markowitz, Sandeep R. Datta, Alex H. Williams, Scott W. Linderman

TL;DR

Researchers have developed WARHMM (Warped Autoregressive Hidden Markov Models), a new framework that disentangles what an animal is doing (discrete "syllables" like grooming or darting) from how they are doing it (continuous variability like speed or vigor). By adding a latent "warping" variable to standard behavioral models, WARHMM reduces the common problem of over-segmenting data and provides a precise way to quantify how factors like drugs or environment change the intensity of specific actions.

The Challenge: Capturing Continuous Variability in Discrete States

In systems neuroscience, a popular way to study behavior is to treat it like language: a sequence of stereotyped "syllables" (e.g., a rear, a left turn, a pause). The standard tool for this is the Autoregressive Hidden Markov Model (ARHMM), which segments video data into these discrete states.

Standard ARHMM pipeline segmenting mouse behavior into discrete syllables.

However, animals aren't robots. A mouse might perform a "dart" very quickly or very slowly. Because standard ARHMMs only understand discrete states, they often mistake a "fast dart" and a "slow dart" for two entirely different behaviors. This leads to over-segmentation, where researchers end up with dozens of redundant behavioral clusters that are difficult to interpret. The gap in the field has been the lack of a model that can account for continuous, structured variability—like movement speed—within a single behavioral category.

Methods: The Warped ARHMM

To address this, the authors introduce the Warped ARHMM (WARHMM). It builds on the standard ARHMM but adds a continuous latent variable, $\tau_t$ (the "warping variable"), which modulates the dynamics of each syllable in real-time.

Graphical models comparing ARHMM and WARHMM, alongside vector fields showing how the warping variable modulates dynamics.

The paper explores two flavors of this warping:

  1. Time-Warped ARHMM (T-WARHMM): This version assumes that variability is primarily about speed. The warping variable acts like a "local clock," linearly scaling the dynamics. If $\tau$ is high, the animal moves through the behavioral trajectory faster; if it's low, the movement slows down.
  2. Gaussian Process-WARHMM (GP-WARHMM): This is a more flexible, non-linear version. It uses Gaussian Processes to allow the warping variable to change the dynamics in complex ways that aren't just about speed, though this comes at the cost of some interpretability.

To make the math tractable, the authors discretized the continuous warping variable onto a fine grid, allowing them to use efficient inference algorithms (like the forward-backward algorithm) typically used for standard HMMs.

Datasets and Validation

The model was validated using several datasets:

  • Synthetic Data: 2D trajectories representing clockwise and counter-clockwise rotations at varying speeds tested if the model could correctly recover both the direction (discrete) and the speed (continuous).
  • The MoSeq Dataset: Depth-camera recordings of 24 freely moving mice. The raw video was pre-processed into the top 10 principal components (PCs) of the mouse's posture.
  • Pharmacological Dataset: Recordings of mice treated with either a saline solution (control) or amphetamine, used to see if the model could detect drug-induced changes in behavioral "vigor."

Key Insights and Results

  • Vigor is a Latent Feature: Instead of just measuring how fast a mouse's center of mass moves (centroid velocity), WARHMM learns an internal "vigor" variable ($\tau$) that is specific to the geometry of the behavior being performed.
  • Efficiency and Reduced Redundancy: A major advantage of WARHMM is that it requires fewer behavioral syllables to describe complex behavior. A WARHMM with only 25 syllables achieved the same performance (in terms of log-likelihood on held-out data) as a standard ARHMM with 40 syllables. In the mouse data, T-WARHMM halved the number of redundant "rear" states compared to the standard model, successfully collapsing fast and slow rears into a single, interpretable category.
  • Drug Sensitivity: The model successfully identified that amphetamine-treated mice move with higher vigor. While a simple speed measurement showed a slight difference between groups, the T-WARHMM's $\tau$ variable showed a significant, syllable-specific shift toward higher speeds across almost all behaviors.
  • Disentangling "What" from "How": The model proves that we can statistically separate the selection of an action from its execution. This is crucial for researchers studying the basal ganglia or dopamine, which are thought to regulate these two aspects of behavior separately.

Limitations and Future Work

While powerful, the current WARHMM framework has limitations. For computational ease, the authors turned the continuous warping variable into a discrete grid. A truly continuous inference method might provide even smoother results. Furthermore, the underlying model still assumes that behavior follows linear autoregressive dynamics; moving toward non-linear dynamics could capture even more nuance. Finally, the current model focuses primarily on speed. Future work could include other continuous axes of variability, such as the "radius" of a turn or the "height" of a posture, to create an even richer map of naturalistic behavior.

Summary updated

Abstract

A core goal in systems neuroscience and neuroethology is to understand how neural circuits generate naturalistic behavior. One foundational idea is that complex naturalistic behavior may be composed of sequences of stereotyped behavioral syllables, which combine to generate rich sequences of actions. To investigate this, a common approach is to use autoregressive hidden Markov models (ARHMMs) to segment video into discrete behavioral syllables. While these approaches have been successful in extracting syllables that are interpretable, they fail to account for other forms of behavioral variability, such as differences in speed, which may be better described as continuous in nature. To overcome these limitations, we introduce a class of warped ARHMMs (WARHMM). As is the case in the ARHMM, behavior is modeled as a mixture of autoregressive dynamics. However, the dynamics under each discrete latent state (i.e. each behavioral syllable) are additionally modulated by a continuous latent ``warping variable.'' We present two versions of warped ARHMM in which the warping variable affects the dynamics of each syllable either linearly or nonlinearly. Using depth-camera recordings of freely moving mice, we demonstrate that the failure of ARHMMs to account for continuous behavioral variability results in duplicate cluster assignments. WARHMM achieves similar performance to the standard ARHMM while using fewer behavioral syllables. Further analysis of behavioral measurements in mice demonstrates that WARHMM identifies structure relating to response vigor.
BibTeX
@inproceedings{costacurta2022distinguishing,
  author = {Julia Costacurta and Lea Duncker and Blue Sheffer and Winthrop Gillis and Caleb Weinreb and Jeffrey E. Markowitz and Sandeep R. Datta and Alex H. Williams and Scott W. Linderman},
  title = {{Distinguishing discrete and continuous behavioral variability using warped autoregressive HMMs}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2022},
  url = {https://openreview.net/forum?id=1quYUVAswP},
}
Figure