Discrete Time Series Generated by Mixtures. I: Correlational and Runs Properties

P. A. Jacobs, P. A. W. LewisJournal of the Royal Statistical Society Series B: Statistical Methodology 1978 · DOI

In the world of time series analysis, we are often spoiled for choice when dealing with continuous data. Standard ARMA (Autoregressive Moving Average) models are the bread and butter of the field. However, when the data is discrete—think of counts of arrivals, binary error logs, or web traffic packets—standard linear models often break down. They struggle to preserve the discrete nature of the data or force us into rigid distributional assumptions.

In their 1977 paper, "Discrete Time Series Generated by Mixtures I," P.A.W. Lewis and P.A. Jacobs introduce a elegant solution: the DARMA (Discrete Mixed Autoregressive-Moving Average) family of models.

TL;DR

The paper introduces a new class of models for discrete time series that allows researchers to specify the marginal distribution (e.g., Poisson, Geometric) and the correlation structure independently. By using probabilistic mixtures instead of linear additions, these models can capture complex, non-Markovian dependencies while remaining mathematically tractable.

What Problem Does This Paper Address?

Modeling dependent sequences of discrete random variables is notoriously difficult. Traditional Markov chains are often "over-parameterized"—the number of parameters explodes as the state space or the order of dependency increases. Furthermore, it is difficult to construct a Markov chain that has a specific desired marginal distribution, like a Poisson distribution, while also exhibiting a specific correlation decay.

Standard ARMA models don't help much here because they rely on summing random variables. Summing two Poisson variables gives you another Poisson, but summing two arbitrary discrete variables usually results in a distribution that is difficult to work with. The authors sought a "parametrically simple" model where the "shape" of the data (the distribution) and the "memory" of the data (the correlation) could be tuned separately.

Methods

The core innovation is the move from linear combinations to probabilistic mixtures.

In a standard AR model, you might say $X_n = \rho X_{n-1} + \epsilon_n$. In the DARMA framework, you don't add terms. Instead, you "pick" between them. For example, in the DAR(1) (Discrete Autoregressive) model, the value $X_n$ is chosen to be:

  • The previous value $X_{n-1}$ with probability $\rho$.
  • A new, independent value $Y_n$ from the underlying distribution with probability $1-\rho$.

Because $X_n$ is always just a "selection" from the sequence of i.i.d. variables $Y_n$, it automatically inherits the marginal distribution of $Y_n$.

The authors generalize this into the DARMA(1, N+1) model, which combines an autoregressive component (looking back at previous states) with a moving average component (looking back at a window of recent independent innovations).

The DARMA model definition as seen in the paper's introduction.

Datasets

The authors evaluate their model using Telephone Error Data originally discussed by Cox and Lewis (1966).

  • Nature: A binary time series where $X_n = 1$ indicates a bit error during transmission over a telephone line, and $X_n = 0$ indicates a successful transmission.
  • Scale: 672 errors out of 1,106,148 transmitted bits.
  • Characteristics: The data is "bursty." Errors tend to cluster, leading to "runs" of zeros (intervals between errors) that are much longer and more skewed than what a simple independent or Markovian model would predict.

Key Insights and Ideas

  • Decoupling Distribution and Correlation: This is the "aha moment." You can pick any discrete distribution for your data—Poisson, Binomial, Negative Binomial—and the correlation equations remain exactly the same. This modularity is a huge win for practitioners.
  • Non-Markovian Memory: While the DAR(1) process is Markovian, the broader DARMA models are generally not. This allows them to model "long-range" dependencies that a simple first-order Markov chain would miss.
  • Run Length Inflation: The authors show that these models naturally produce longer "runs" of the same value compared to independent sequences. This makes them ideal for modeling "bursty" phenomena like network errors or periods of high/low activity.
  • Transformation Invariance: If you apply a function to every element in a DARMA sequence, the correlation structure of the transformed sequence remains identical to the original.

Key Results

The paper provides closed-form expressions for the autocorrelation functions and power spectra of these processes.

In the application to telephone error data, the authors demonstrate that the binary DARMA(1,1) model can account for the highly skewed distribution of error-free intervals. They show that by tuning the mixture parameters ($\beta$ and $\rho$), they can match the observed mean run lengths of the data, which were significantly higher than what an independence assumption would suggest.

The following table from the paper illustrates how the expected length of a run of zeros ($E(T_0)$) changes as the model parameters are varied, highlighting the model's flexibility.

Table 1: Expected length of runs of zeros in a binary DARMA(1,1) process.

Limitations and Future Work

  • Time Reversibility: The authors note that these models are generally not time-reversible. If you play the sequence backward, it doesn't look the same statistically as the forward sequence.
  • Continuous Data Drawback: While the model can be used for continuous data, the authors admit a major drawback: because the model works by "picking" previous values, a realization of a continuous DARMA process would contain many identical values (runs), which is physically unrealistic for most continuous phenomena.
  • Parameter Estimation: This paper focuses on the properties of the models. The "systematic" estimation of parameters from raw data is left for "Part II" and subsequent research.

Summary updated

Abstract

Summary A broad but parametrically simple model for a stationary sequence of dependent discrete random variables is given and several submodels are discussed. The structure of the model is specified by the marginal distribution of the random variables and several other parameters. The sequence of random variables is formed by a probabilistic linear combination of i.i.d. discrete random variables and is in general not Markovian. Second-order joint moments and spectra are obtained for the model, as well as some properties for the lengths of runs. The special case of a process in which the variables take on only two values is considered; this binary process is useful as a model for the counting process in a discrete-time point process.
BibTeX
@article{jacobs1978discrete,
  author = {P. A. Jacobs and P. A. W. Lewis},
  title = {{Discrete Time Series Generated by Mixtures. I: Correlational and Runs Properties}},
  journal = {Journal of the Royal Statistical Society Series B: Statistical Methodology},
  year = {1978},
  doi = {10.1111/j.2517-6161.1978.tb01653.x},
}
Figure