Modeling Video Dynamics with Linear Dynamical Systems

22 concepts15 papers1 proposal

When we look at a video of a flickering flame, a rushing river, or leaves swaying in the wind, we easily recognize the scene despite the chaotic, non-rigid movement of individual pixels. In computer vision, these phenomena are known as dynamic textures.

Traditional visual processing methods often struggle with dynamic textures because they treat video either as a sequence of isolated static frames or as a collection of discrete pixel trajectories (such as optical flow). A flame does not have a fixed spatial template or a clean tracking point; its appearance is inherently stochastic and continuous.

To capture the essence of this motion, a powerful framework models video sequences as the outputs of Linear Dynamical Systems (LDS). Instead of tracking what pixel goes where, this approach models the underlying physical or statistical process that generates the pixels.


The Core Mechanism: State-Space Representation

The intuition behind using an LDS for video modeling relies on a simple generative assumption: the complex, high-dimensional visual appearance of a video frame is driven by a simpler, low-dimensional hidden state that evolves over time.

Mathematically, this is expressed through two coupled equations (often framed as an Autoregressive Moving Average (ARMA) or Gauss-Markov process):

  1. The State Transition Equation: $$x_{t+1} = Ax_t + v_t$$ The latent state $x_t$ represents the internal dynamics of the system at time $t$. The matrix $A$ defines how this state evolves linearly into the next frame's state, while $v_t$ represents Gaussian process noise that injects stochastic variability (e.g., the unpredictability of smoke).

  2. The Observation Equation: $$y_t = Cx_t + w_t$$ The vector $y_t$ represents the observed pixel intensities or feature representations of the video frame at time $t$. The matrix $C$ is a linear projection that maps the low-dimensional hidden state back into the high-dimensional image space, acting as the visual appearance template. The term $w_t$ represents observation noise.

By fitting this model to a video sequence, we decouple the video into two distinct components: appearance (captured by $C$) and dynamics (captured by $A$). As shown by Doretto et al., this allows us to characterize video regions based on their temporal patterns rather than static spatial features alone.


Why It Works: The Intuition of System Signatures

Representing a video as an LDS offers several fundamental advantages over traditional frame-by-frame representations:

1. Invariance to Sequence Length and Alignment

Comparing two videos of different lengths normally requires computationally expensive temporal warping. However, if both videos are modeled as linear time-invariant systems, we can compare the underlying system parameters ($A$ and $C$) directly. For instance, Lo Presti et al. demonstrated that by constructing Hankel matrices from the system outputs, one can extract a signature of movement that is robust to varying sequence lengths. This makes it possible to recognize facial expressions or detect pain without needing explicit temporal alignment.

2. Robustness to Noise

Because the pixel observations $y_t$ are projected from a low-dimensional state space, the model inherently filters out high-frequency spatial and temporal noise. As noted by Chan et al., learning the underlying probability distribution of a texture provides superior robustness to pixel-level noise compared to discriminative methods like optical flow, enabling the clean segmentation of overlapping stochastic phenomena (such as fire burning in front of moving water).

3. Global Dynamics over Discrete Transitions

While models like Hidden Markov Models (HMMs) rely on transitions between discrete states, complex real-world events are often continuous. Bhattacharya et al. showed that capturing global dynamics through matrix decomposition of linear systems offers a significant performance advantage over bag-of-features and HMM approaches for complex event retrieval, because it models the continuous flow of information.


Applications Across Computer Vision

The ability to treat motion as a predictable, linear state-space transition has driven breakthroughs in several core vision tasks.

Background Subtraction in Dynamic Scenes

Standard background subtraction assumes a static camera and a stationary background. When the background contains dynamic textures—like waving curtains or rain—pixels change constantly.

By modeling the background as an LDS, the chaotic motion becomes a predictable state transition. Foreground objects can then be identified as statistical outliers that violate the system's learned dynamics.

  • Zhong et al. utilized a Robust Kalman Filter to isolate foreground objects from ARMA-modeled backgrounds.
  • Vasconcelos et al. extended this concept to highly dynamic scenes (like surfers on crashing waves) by modeling spatiotemporal video patches as local dynamic textures, allowing for unsupervised saliency detection even from a moving camera.
  • Mumtaz introduced a Foreground-Background Mixture (FBM) model that jointly estimates the stochastic background and segments moving objects without requiring clean training videos.
  • Lin et al. organized scenes into "video bricks" modeled via ARMA processes to maintain robust background subtraction under sudden illumination changes.
[ Observed Video Frame ] 
       │
       ▼
[ Robust Kalman Filter / LDS Model ] ──► Matches Learned Dynamics?
       │                                       │
       ├───────────────────► YES ──────────────┴──► Label as Background
       │
       └───────────────────► NO  ─────────────────► Label as Foreground Object

Tracking Non-Rigid Objects via "Dynamic Templates"

Tracking an object typically requires a consistent visual template. But how do you track something that changes shape and appearance constantly, like smoke or a running person?

The authors in Chaudhry et al. and Chaudhry et al. solved this by proposing dynamic templates. Instead of tracking a static image patch, the system tracks the parameters of an LDS. The target is defined by how its appearance evolves over time, allowing for simultaneous tracking and category recognition of non-rigid and articulated shapes.

Classification and Action Recognition

Beyond pixel intensities, the LDS framework can capture the evolution of high-level features.

  • Statistical Classifiers: To classify dynamic textures using discriminative classifiers like Support Vector Machines (SVMs), Chan et al. derived a probabilistic kernel based on the Kullback-Leibler (KL) divergence between different auto-regressive visual processes.
  • Deep Feature Dynamics: In modern architectures, Li et al. introduced VLAD³, a hierarchical representation that uses LDS to model the temporal flow of deep features across multiple time scales, proving that explicit mathematical modeling of state transitions outperforms simple feature pooling for sequence-heavy tasks.
  • Unsupervised Signal Segmentation: This framework is not limited to video; Yin et al. applied LDS to raw sensor signals to capture local temporal trends, automatically segmenting continuous signals into atomic motion patterns for activity recognition.

Generative Modeling and Shape Analysis

Because an LDS is fundamentally a generative time-series model, it can be used to evaluate the physical plausibility of sequences. For example, Yousefi et al. combined Dirichlet Process Mixtures with LDS to create a fully Bayesian framework that automatically segments an unknown number of overlapping dynamic textures. In medical imaging, Khan et al. leveraged the predictive power of an LDS to evaluate and validate spatiotemporal cardiac shape models, proving that the system can accurately reconstruct and generalize complex organ motion over time.

Key Foundational Work

Dynamic Texture Segmentation (2003) — contributed 1 concept elements to this direction.
Activity Recognition through Goal-Based Segmentation. (2005) — contributed 1 concept elements to this direction.
Probabilistic Kernels for the Classification of Auto-Regressive Visual Processes (2005) — contributed 1 concept elements to this direction.
History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 15 papers
  2. Version 1 · 2026-10-04 · 20 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_linear_time_invariant_system_modeling_aa7a670b,
  title = {linear time-invariant system modeling},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/linear-time-invariant-system-modeling/rdr_aa7a670b3864409aaf16854d1cc74755},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 2, frozen text):

@misc{lacuna_linear_time_invariant_system_modeling_aa7a670b_v2,
  title = {linear time-invariant system modeling},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/linear-time-invariant-system-modeling/rdr_aa7a670b3864409aaf16854d1cc74755/v/2},
  note = {Version 2, frozen 2026-10-04. Accessed 2026-11-21},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake