When a model has to discover how many hidden regimes a time series contains, it faces a choice every time the data look unfamiliar. One option is that the system has entered a new regime, with its own dwell time and its own successors. The other is that the system is still in the same regime and is simply showing one of that regime's several faces. A speaker who alternates between voiced and breathy speech is still the same speaker. A worm's forward-crawling neural state can show two calcium signatures without becoming a different behavior. Current nonparametric sequence models resolve this choice through their priors rather than through the data. The result is a familiar failure: states multiply to cover emission variability, segments get cut short, and the inferred number of states depends on hyperparameter tuning.
The core idea of this note is that time, not appearance, should decide. A regime is something that predicts the future: how long the current segment will last and what comes next. A mode of a regime is something that does not. We propose a Bayesian nonparametric hidden semi-Markov model that has two parts. The first is explicit, observation-dependent durations for an unbounded set of states. The second is a Dirichlet-process mixture inside each state's emission distribution, whose mode labels are drawn independently at each time step and so carry no information about the clock or the next state. We claim that each component repairs the other's characteristic failure. Explicit durations with a minimum dwell make the state-versus-mode decision identifiable. Within-state mixtures stop multimodal emissions from fragmenting segments and corrupting the duration estimates.
Background and the gap
Infinite hidden Markov models replace a fixed number of states $K$ with a hierarchical Dirichlet process (HDP), a prior that offers a shared, unbounded pool of states which every transition row can reuse. This line of work runs from The Infinite Hidden Markov Model to the sticky variant of Fox et al.. The sticky model adds a self-transition bonus to prevent rapid switching between near-duplicate states. Its speaker-diarization version, A sticky HDP-HMM with application to speaker diarization, already places a Dirichlet-process Gaussian mixture inside each state, and it reports a diarization error rate of 17.84% on NIST Rich Transcription meetings against 23.91% without stickiness. That paper supplies half of our method. Its durations, however, remain geometric, and geometric durations are exactly what make state and mode interchangeable. A mode that happens to persist for a few frames looks the same as a short-lived state, because both produce memoryless run lengths.
Three later papers relax the duration side but give up flexible emissions. Disentangled sticky HDP-HMM gives each state its own stickiness. The Recurrent Sticky HDP-HMM lets stickiness depend on the previous observation. RED-HDP-HMM introduces observation-dependent negative-binomial durations and a "nonloopy" constraint that forbids self-transitions when a duration expires. It reports 0.899 accuracy on the honeybee dance data, against 0.873 for the sticky HDP-HMM and 0.853 for the recurrent sticky model, and it gains 4.5 to 10.4 percentage points over sticky models on C. elegans calcium imaging. RED-HDP-HMM supplies the other half of our method, but its emissions are single Gaussian or autoregressive components. The authors themselves flag that switching models cannot tell whether a shift in the data comes from the transition logic or from the emission model.
HDP-Flow takes a different route. It conditions a normalizing flow, a flexible neural density model, on the state and the dwell time. This gains flexibility but loses any explicit notion of a mode, and the authors report that state granularity is highly sensitive to the Dirichlet-process priors. HDP-GPC separates shape from timing using Gaussian-process warps, but it works only on pre-segmented one-dimensional series. None of these papers states a criterion for when new data should open a new state versus a new mode within an existing state. That missing criterion is the gap we target.
Core Claim
Each state $j$ has three components. The first is a transition row with no self-loops, $\pi_j$, drawn from an HDP. The second is a duration law $p(d \mid j, y_{\text{past}})$, a negative binomial with a logistic link on recent observations, following RED-HDP-HMM. The third is an emission mixture $G_j = \sum_m w_{jm}\, \mathcal{N}(\mu_{jm}, \Sigma_{jm})$ with $w_j \sim \text{GEM}(\eta)$, where GEM is the stick-breaking distribution of Dirichlet-process weights. All emission atoms come from a shared base measure. Within a segment of state $j$, each observation draws its own mode label $c_t \sim w_j$ independently. This design builds in the conditional independence
$$ c_t \;\perp\; (r_t,\, z_{\text{next}}) \;\mid\; z_t , $$
Here $r_t$ is the remaining duration of the current segment and $z_{\text{next}}$ is the state that follows it.
First claim. When the minimum dwell exceeds the typical run length of a within-state mode, a pattern that violates this independence can be explained more cheaply by a new state than by a new mode, and the reverse holds for a pattern that satisfies it. We will make this precise as a sufficient condition on synthetic generative families, comparing marginal likelihoods of the two explanations as sequence length grows. We will then check it empirically.
Second claim. On real data the combination beats both parents. It should beat the sticky HDP-HMM with mixture emissions on segment structure, and beat RED-HDP-HMM on state-count accuracy wherever emissions are multimodal.
Research Program
The model combines two existing Gibbs samplers, so the first deliverable is a weak-limit blocked sampler. The state-and-countdown forward-backward pass and the Pólya-Gamma updates for duration regressions come from RED-HDP-HMM. Pólya-Gamma augmentation is a trick that makes logistic regressions conditionally Gaussian. The per-state mixture updates come from Fox et al. With state truncation $L$, maximum duration $D_{\max}$ and mode truncation $M$, the cost per sweep is $O(TL(L + D_{\max}) + TLM)$, so the mode layer adds only a linear term.
The second deliverable is a pair of split-merge moves that use the identity above directly. One move proposes to promote a mode to its own state, and the other proposes to demote a state into a mode of a neighboring state. Each proposal is weighted by an estimate of the conditional mutual information $I(c_t; r_t, z_{\text{next}} \mid z_t)$ computed from current samples. The same quantity serves as a posterior-predictive diagnostic. A fitted model whose modes carry clock information has misallocated novelty.
The third deliverable is the identifiability analysis. We will build synthetic families in which ground-truth states have $1$ to $4$ emission modes with known mixing, durations with controlled minimum dwell, and controlled overlap between modes of different states. Across these settings we will map where state and mode recovery succeeds or fails, and we will test whether the failure boundary tracks the ratio of minimum dwell to mode run length, as the first claim predicts. Scaling to variational inference is out of scope. We keep MCMC so that approximation error does not confound the identifiability question; Zhang et al. note that mean-field over-counts states.
Baselines
The naive reference is a finite hidden semi-Markov model with single Gaussian emissions. Its $K$ is chosen by BIC over a sweep, in the spirit of the model-selection approaches in State Space Construction for Behavior Acquisition and Data-Driven Learning of the Number of States. It shows what the nonparametric machinery buys when the data are batch and stationary.
The first strong existing method is RED-HDP-HMM. It holds the best reported honeybee accuracy in the file, 0.899, and the largest reported C. elegans gains. It is the direct competitor on segmentation, and to beat it we must exceed 0.899 on honeybee. On NIST diarization, the sticky HDP-HMM with Dirichlet-process mixture emissions is the strongest reported system in the file, at 17.84% DER, and beating it means lower DER on the same 21 meetings. Because our feature pipeline will differ from the 2009/2011 papers, we will rerun both parents in our own code, with the same features, truncation levels, base measures, iteration counts, and an equal-size hyperparameter grid for each method. We will report the published numbers only as anchors.
HDP-Flow is the flexible-emission competitor, and we will run it on HAR70+, where the file reports that it scaled but gives no segmentation number. No reported result was found on Lacuna for HDP-Flow segmentation accuracy, so we will produce one.
Two ablations isolate the key ingredient. The first swaps our explicit durations for geometric ones while keeping the mixtures, which tests the first claim. The second swaps the Dirichlet-process emissions for a fixed three-component mixture per state, which tests whether nonparametric emissions matter at all.
Why not just…?
Why not just use RED-HDP-HMM with a fixed small Gaussian mixture per state? This is the most serious threat. If most real states have two or three modes, a fixed mixture may match ours, and the nonparametric layer would only add sampler cost. The second ablation measures this directly, and we will report it honestly. Our expectation is that fixed mixtures fail in two places: states whose mode count varies, as with speakers who differ in vocal variability, and data where unused components are absorbed as spurious duplicate states. If the ablation ties our model on all benchmarks, the contribution narrows to the duration-based identifiability result and the diagnostic, and we will say so.
Why not just fit the sticky HDP-HMM with mixture emissions and merge states afterward whose transitions look alike? Post-hoc merging cannot fix segment boundaries that were placed wrongly during inference. Geometric durations also give the merge rule no clock-based evidence to work with. The geometric-duration ablation, given the same merge step, will show whether merging recovers RED-level segmentation. The honeybee gap between 0.873 and 0.899 suggests duration modeling contributes something that merging alone would not.
Why not just use HDP-Flow's neural emissions? Flows can absorb multimodality, but they give no explicit mode variable, so the state-versus-mode decision stays implicit and prior-driven. HDP-Flow's own reported sensitivity of granularity to the priors is the symptom we target. If HDP-Flow matches our state-count accuracy on HAR70+ and synthetic data at comparable compute, then the explicit mode layer would matter only for interpretability.
Why not just sweep $K$ with BIC? Such a sweep needs batch data and has no mechanism for a regime that appears partway through. On Extended NASCAR, irSLDS showed that fixed-$K$ rSLDS forced new dynamics into old categories. A BIC sweep also selects a single $K$ and does not separate states from modes.
Proposed Experiments
The decisive synthetic check comes first. We vary the ratio of minimum dwell to mode run length and measure adjusted Rand index on states, error in the number of states, and error in modes per state. If the first claim holds, our model's error should collapse once the ratio exceeds one, and the geometric-duration ablation's error should not.
Real-data evaluation then uses four benchmarks. On NIST Rich Transcription we report DER and inferred speaker count. On honeybee dance we report accuracy and weighted F1. On C. elegans calcium imaging we report segmentation accuracy against annotated behavior. On HAR70+ we report segmentation accuracy and state-count error. Across all four we also report how stable the inferred state count is over a fixed grid of concentration hyperparameters. The core prediction is that this stability improves over RED-HDP-HMM and HDP-Flow, because modes absorb emission variability that would otherwise spawn states.
The final check is the conditional-mutual-information diagnostic on held-out fits. Correct models should show near-zero $I(c_t; r_t, z_{\text{next}} \mid z_t)$. We will measure how often baselines that are given a mode layer post hoc violate it.
References
- The Infinite Hidden Markov Model
- The Sticky HDP-HMM: Bayesian Nonparametric Hidden Markov Models with Persistent States
- A sticky HDP-HMM with application to speaker diarization
- Disentangled Sticky Hierarchical Dirichlet Process Hidden Markov Model
- The Recurrent Sticky Hierarchical Dirichlet Process Hidden Markov Model
- RED-HDP-HMM: Observation-Dependent Durations for Bayesian Nonparametric Sequential Models
- HDP-Flow: Generalizable Bayesian Nonparametric Model for Time Series State Discovery
- Bayesian Nonparametric Dynamical Clustering of Time Series
- Stochastic Variational Inference for the HDP-HMM
- State Space Construction for Behavior Acquisition in Multi Agent Environments with Vision and Action
- Data-driven learning of the number of states in multi-state autoregressive models
- Parsing neural dynamics with infinite recurrent switching linear dynamical systems