Distinguishing discrete and continuous behavioral variability using warped autoregressive HMMs

Julia C. Costacurta, Lea Duncker, Blue Sheffer, Winthrop F. Gillis, Caleb Weinreb, Jeffrey E. Markowitz, Sandeep Robert Datta, Alex H. Williams, Scott W. Linderman

Abstract

A core goal in systems neuroscience and neuroethology is to understand how neural circuits generate naturalistic behavior. One foundational idea is that complex naturalistic behavior may be composed of sequences of stereotyped behavioral syllables, which combine to generate rich sequences of actions. To investigate this, a common approach is to use autoregressive hidden Markov models (ARHMMs) to segment video into discrete behavioral syllables. While these approaches have been successful in extracting syllables that are interpretable, they fail to account for other forms of behavioral variability, such as differences in speed, which may be better described as continuous in nature. To overcome these limitations, we introduce a class of warped ARHMMs (WARHMM). As is the case in the ARHMM, behavior is modeled as a mixture of autoregressive dynamics. However, the dynamics under each discrete latent state (i.e. each behavioral syllable) are additionally modulated by a continuous latent “warping variable.” We present two versions of warped ARHMM in which the warping variable affects the dynamics of each syllable either linearly or nonlinearly. Using depth-camera recordings of freely moving mice, we demonstrate that the failure of ARHMMs to account for continuous behavioral variability results in duplicate cluster assignments. WARHMM achieves similar performance to the standard ARHMM while using fewer behavioral syllables. Further analysis of behavioral measurements in mice demonstrates that WARHMM identifies structure relating to response vigor.

Introduction: The Challenge of Behavioral Variability

In systems neuroscience, a popular theory suggests that complex animal behavior is composed of "behavioral syllables"—short, stereotyped movements such as a "rear," a "groom," or a "dart." To identify these automatically, researchers typically employ Autoregressive Hidden Markov Models (ARHMMs) to segment high-dimensional video data into discrete states.

Overview of the behavioral segmentation pipeline: raw video frames are reduced via PCA, segmented into time series, and mapped to discrete syllables.

However, standard ARHMMs face a significant limitation: they struggle to account for continuous variability within a single type of action. For instance, a mouse may perform a "dart" at varying speeds. Because standard models only recognize discrete states, they often categorize a fast dart and a slow dart as entirely different behaviors. This leads to over-segmentation, where the model generates redundant syllables that represent the same action performed with different intensities. This redundancy complicates efforts to relate behavior to neural activity or to study the effects of pharmacological interventions.

Methods: Introducing the Warped ARHMM

To address this, we developed WARHMM, which extends the standard ARHMM by incorporating a continuous latent variable, $\tau_t$, known as the "warping variable." In this framework, while the discrete state $z_t$ identifies the type of behavior, $\tau_t$ modulates the specific dynamics of that behavior.

Graphical model comparison between standard ARHMM and Warped ARHMM, showing how the warping variable modulates dynamics across different states.

We explored two primary implementations of this warping mechanism:

  1. Time-Warped ARHMM (T-WARHMM): This version assumes a linear relationship where $\tau$ acts as a "log step-size," effectively speeding up or slowing down the behavioral "clock." This is highly interpretable and maps directly to the biological concept of movement vigor.
  2. Gaussian Process WARHMM (GP-WARHMM): This version utilizes Gaussian Processes to allow $\tau$ to modulate behavior nonlinearly. While less intuitive than T-WARHMM, it offers greater flexibility to capture complex variations beyond simple speed changes.

To ensure computational efficiency, the continuous warping variable is discretized onto a fine grid, allowing for the use of standard inference algorithms like Expectation-Maximization.

Key Insights and Results

Efficiency and Reduced Over-segmentation

WARHMM demonstrates superior efficiency by achieving the same predictive power (log-likelihood) as standard ARHMMs while using significantly fewer discrete states. For example, a T-WARHMM with 25 states performs comparably to a standard ARHMM with 40 states. Expert observation confirmed that WARHMM successfully groups "fast" and "slow" versions of the same behavior into single states, whereas standard models split them into redundant categories.

Vigor as a Latent State

The inferred warping variable $\tau$ correlates strongly with physical metrics such as centroid velocity. This suggests that "vigor" is a fundamental internal state that the model can extract directly from posture data. On held-out test data, both WARHMM variants consistently outperformed the standard ARHMM in log-likelihood across various state counts.

Pharmacological Detection

Using a dataset of mice treated with amphetamine versus saline, WARHMM identified a significant rightward shift in the vigor distribution for treated mice across nearly all behavioral syllables ($p < 0.05$). This indicates that the drug does not necessarily introduce new behaviors, but rather shifts the style (vigor) of the existing behavioral repertoire.

Conclusion and Future Directions

WARHMM provides a robust framework for disentangling the "what" (discrete actions) from the "how" (continuous style) of animal behavior. While the current model uses a discretized grid for $\tau$ and assumes linear underlying dynamics, future iterations could incorporate truly continuous inference or nonlinear dynamics (e.g., via Variational Autoencoders). Furthermore, the warping variable could be expanded to include other continuous axes of movement, such as turning radius or body height, to further refine our understanding of naturalistic behavior.

Summary updated

Abstract

Abstract A core goal in systems neuroscience and neuroethology is to understand how neural circuits generate naturalistic behavior. One foundational idea is that complex naturalistic behavior may be composed of sequences of stereotyped behavioral syllables, which combine to generate rich sequences of actions. To investigate this, a common approach is to use autoregressive hidden Markov models (ARHMMs) to segment video into discrete behavioral syllables. While these approaches have been successful in extracting syllables that are interpretable, they fail to account for other forms of behavioral variability, such as differences in speed, which may be better described as continuous in nature. To overcome these limitations, we introduce a class of warped ARHMMs (WARHMM). As is the case in the ARHMM, behavior is modeled as a mixture of autoregressive dynamics. However, the dynamics under each discrete latent state (i.e. each behavioral syllable) are additionally modulated by a continuous latent “warping variable.” We present two versions of warped ARHMM in which the warping variable affects the dynamics of each syllable either linearly or nonlinearly. Using depth-camera recordings of freely moving mice, we demonstrate that the failure of ARHMMs to account for continuous behavioral variability results in duplicate cluster assignments. WARHMM achieves similar performance to the standard ARHMM while using fewer behavioral syllables. Further analysis of behavioral measurements in mice demonstrates that WARHMM identifies structure relating to response vigor.
BibTeX
@inproceedings{costacurta2022distinguishing,
  author = {Julia C. Costacurta and Lea Duncker and Blue Sheffer and Winthrop F. Gillis and Caleb Weinreb and Jeffrey E. Markowitz and Sandeep Robert Datta and Alex H. Williams and Scott W. Linderman},
  title = {{Distinguishing discrete and continuous behavioral variability using warped autoregressive HMMs}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2022},
  url = {https://openreview.net/forum?id=1quYUVAswP},
}
Figure