Fisher Kernels That Keep Explaining Away

A long-standing way to get the best of generative and discriminative learning is the Fisher kernel. You fit a probabilistic model of the data, then describe each sequence by the gradient of its log-likelihood with respect to the model's parameters: how the model would need to change to fit this sequence better. A discriminative classifier, such as a support vector machine, then works on those gradient vectors. The method has so far been built on hidden Markov models (HMMs) with a single categorical hidden state, where the needed posterior is exact and cheap. Many real sequences come from several processes running at once, such as two body parts, two dancers, or locomotion alongside a hand activity. These are better modeled by a factorial HMM, which has several parallel hidden chains whose effects add up in one observation. In such models the chains are independent before seeing data but become coupled after seeing it, because they compete to explain the same observation. This is the "explaining away" effect. The standard fix is a mean-field approximation, which treats the chains as independent in the posterior and so deletes that coupling.

This note makes a narrow, checkable claim. For a factorial HMM with linear-Gaussian emissions, the exact Fisher score with respect to the emission weights equals the mean-field score plus one extra term. That term is the posterior cross-chain covariance, the explaining-away signal itself, passed through the emission weights. Mean field sets the term to zero by construction. We propose to compute it with posteriors that keep the coupling, and to test whether it carries class information. We expect it to matter when classes differ in how their underlying processes compete for the same sensor dimensions. If it does not help, practitioners can safely use mean field for feature extraction, and that is also worth knowing.

Core Claim

Factorial Hidden Markov Models defines observations as $y_t = \sum_m W^{(m)} s_t^{(m)} + \epsilon_t$ with noise covariance $C$. Here $s_t^{(m)}$ is a one-hot state vector for chain $m$ at time $t$, and $W^{(m)}$ is that chain's emission matrix. Differentiating the expected complete-data log-likelihood gives the Fisher score for chain $m$'s emission weights:

$$U_{W^{(m)}} = C^{-1}\sum_t \Big( y_t\,\mathbb{E}[s_t^{(m)}]^\top - \sum_{n} W^{(n)}\,\mathbb{E}[s_t^{(n)} s_t^{(m)\top}] \Big).$$

Writing each cross moment as a product of means plus a covariance splits this exactly:

$$U_{W^{(m)}} = U^{\text{MF}}_{W^{(m)}} \;-\; C^{-1}\sum_t \sum_{n\neq m} W^{(n)}\,\mathrm{Cov}\big(s_t^{(n)}, s_t^{(m)}\big).$$

In this equation $U^{\text{MF}}$ is the score obtained with the same marginals but independent chains. The covariance terms are zero under a fully factorized posterior. They are large exactly when two chains' emission columns overlap, so that turning one chain "on" makes the other unnecessary. The score with respect to transition parameters depends only on single-chain pairwise marginals, so explaining away enters it only indirectly, through the marginals.

The proposal is that this correction term is a usable discriminative feature, and that keeping it restores some of what Exploiting Generative Models in Discriminative Classifiers guarantees. That paper shows a Fisher-kernel classifier asymptotically does at least as well as the generative model's own maximum-a-posteriori rule. The guarantee assumes exact scores. With an approximate posterior $q$, the features become gradients of the variational lower bound, and whether the guarantee survives is untested. The research file lists this exact question, the Fisher score over coupled factorial posteriors, as open, with no paper spanning all three directions.

Positioning Against Prior Work

The proposal builds on three bodies of work, each in a different role. The Fisher kernel paper above supplies the method but uses only single-state HMMs. Factorial Hidden Markov Models supplies the generative model and its mean-field and Gibbs approximations, but evaluates only likelihood and never extracts features for classification.

Two discriminative models with distributed hidden states motivate the work by showing that factored latent states help classification. Multivariate Time-Series Classification Using the Hidden-Unit Logistic Model (HULM) uses chains of binary hidden units. Factored Latent-Dynamic Conditional Random Fields for Single and Multi-label Sequence Modeling (FLDCRF) uses several interacting hidden layers. Both condition on the observations instead of generating them, so explaining away never appears in their structure. HULM in particular assumes the hidden units are conditionally independent given input and label in order to keep inference linear-time. Our proposal keeps the directed, generative direction those models give up. That lets us train on unlabeled sequences and lets the competition between causes show up in the features.

Three methods for coupled posteriors supply our inference machinery. Faithful Inversion of Generative Models for Effective Amortized Inference (NaMI) works out which edges a recognition network must contain to represent the true posterior. Applied to a factorial HMM, it says all chains at a time step must be linked, because they share $y_t$. Structured Recognition for Generative Models with Explaining Away (SRVAE) shows that recognition networks producing joint potentials, instead of per-variable ones, give tighter bounds. Hamming Ball Auxiliary Sampling for Factorial Hidden Markov Models gives a sampler for coupled factorial posteriors. All three evaluate only generative quality. None asks whether the recovered coupling helps a classifier, which is the question here.

Products of Hidden Markov Models: It Takes N>1 to Tango constrains the design. It removes explaining away by switching to an undirected product of HMMs. That model has an intractable normalizer, which would enter the Fisher score through the gradient of the log normalizer. It is therefore a poor host for Fisher kernels, and we do not build on it.

Research Program

We fit a factorial HMM with $d$ chains of $K$ states to unlabeled training sequences, using all classes pooled. We also fit a per-class variant, as in the original Fisher-kernel recipe. For each sequence we then compute the score vector over emission weights and transition matrices, from the posterior moments each estimator provides.

We use three estimators of the moments, which also serve as checks on one another. The first is exact forward-backward on the product state space. This is feasible for small configurations: with $d=3$ and $K=3$ there are 27 joint states, and the cost per step, which scales as $K^{2d}$, is 729 operations. The second is an amortized recognition network following NaMI's edge set. It outputs a joint potential over all chains at each time step, in the spirit of SRVAE, and is combined with forward-backward along time. The third is Hamming-ball sampling, for larger $d$ where exact computation is too costly.

The mean-field score is computed from the same fitted model, so that the only difference between conditions is the posterior covariance term. Scores are whitened with a diagonal Fisher information estimate, which avoids inverting the full matrix. A linear support vector machine and an RBF-kernel support vector machine are then trained on the whitened scores. We include the RBF version because Jaakkola and Haussler note that linear separators in score space are often insufficient.

The analysis has two parts. First, for the decomposition above we report how much of the score's variance and of the classifier's weight falls on the covariance term. Second, we measure how far the variational-score classifier falls short of the generative maximum-a-posteriori rule computed from the same model. This is a direct empirical test of whether Theorem 1 holds approximately under mean-field versus coupled posteriors.

Baselines

The naive reference point is the "Naive Logistic" model in the five-benchmark suite of Multivariate Time-Series Classification Using the Hidden-Unit Logistic Model, which is logistic regression on time-pooled features. The file gives no individual figure for it, so we rerun it on our splits. Next comes the classic Fisher kernel from a single-state HMM, following Exploiting Generative Models in Discriminative Classifiers. We run it twice: once with $K^d$ states, matching the number of configurations, and once with the state count chosen to match the factorial model's parameter count. This tells us whether factoring the state matters at all before we ask whether explaining away matters.

The strongest existing competitor is HULM, because it is the best distributed-state sequence classifier in the file. Pei et al. report 6.44% generalization error on CK+ facial-expression classification and an 18.75% average error across the five benchmarks (online handwriting, Arabic spoken digits, CK+, MSR Action 3D, MSR Daily Activity 3D). For comparison, the file reports 11.04% error for HCRF on CK+. It gives no individual figure for the Fisher-kernel-learning baseline in that suite, so no reported result was found on Lacuna for that number. On UCI Opportunity and Gesture Phase, FLDCRF is the strongest competitor. The file reports only relative gains over LSTMs (about 2% and 4%), not absolute accuracies, so we rerun FLDCRF on our splits. An LSTM trained with cross-entropy is included because it is the default a reviewer would reach for.

For fairness, all methods use the same train and test splits as Pei et al. where those exist. HULM's numbers are reproduced before any comparison is made. Every method gets the same hyperparameter search budget, measured in trials, and is selected on the same validation split. Hidden capacity is matched across models: the total number of hidden configurations is matched as closely as possible between HULM's binary units and the factorial model's chains.

The key ablation is the mean-field Fisher score from the identical fitted model. A second ablation feeds the pooled posterior moments themselves, the means and cross-chain covariances, directly to the same support vector machine without the Fisher-score construction. This separates "the coupling statistics are informative" from "the score geometry is what helps."

Beating HULM means lower error than 6.44% on CK+, or a lower five-suite average than 18.75%, under reproduced splits. If we fail to beat HULM with all labels, we still count it as a win if the method beats HULM in the low-label regime. There the factorial model is fit on all sequences without labels, and the classifier sees only 5–20% of the labels. Beating the mean-field ablation means a statistically reliable gain across seeds, on the benchmarks where the posterior covariance is non-negligible.

Why not just…?

Why not just use the mean-field Fisher score? This is the real threat. If the covariance term is small or redundant with the marginals, mean field gives the same features at lower cost. Generative evidence suggests otherwise. Ignoring the coupling worsens reconstruction and likelihood in Structured Recognition for Generative Models with Explaining Away, and it limits expressivity in Discrete Variational Autoencoders. Neither result says anything about classification, however. The synthetic dose-response experiment below settles the question. We will report it even if the answer is "mean field suffices."

Why not just train HULM or FLDCRF end to end? These models avoid explaining away by conditioning on the input, and they are strong. With all labels on CK+, HULM may simply win. Our case rests on two regimes they cannot exploit. The first is unlabeled data, since only a generative model can use it. The second is classes that differ in how causes compete for shared sensor dimensions, which a conditional model sees only indirectly. The low-label curves and the overlap experiment test both regimes. If HULM matches us there too, the proposal's practical value is limited to interpretability, and we will say so.

Why not just feed posterior statistics, or raw sequences, to a neural classifier? An LSTM could learn coupling-like features from raw data. FLDCRF's authors also report that LSTMs vary across runs and generalize inconsistently from validation to test. The moment-feature ablation answers the more pointed version of this question. If pooled cross-covariances fed to a support vector machine perform as well as the full corrected score, the contribution is "explaining-away statistics as features," not Fisher geometry, and we would reframe the work accordingly.

Proposed Experiments

The decisive test is synthetic. We generate two-class data from factorial HMMs with $d=3$ and $K=3$. The classes differ in their emission matrices, and a single knob sets the angle between different chains' emission columns. When columns are orthogonal, explaining away vanishes and mean field is nearly exact. As the overlap grows, the chains compete more.

The prediction is a dose-response curve. The accuracy gap between the exact score and the mean-field score should be near zero at orthogonality and should grow with overlap. The structured and Hamming-ball estimators should track the exact score. The same curve, measured as the gap from the generative maximum-a-posteriori rule, tests whether Theorem 1 holds approximately. A flat curve falsifies the core claim.

On real data we evaluate on the five-benchmark suite, with CK+ as the headline because both error and Action Unit AUC are reported there (HULM 0.9253 versus CRF 0.8647). We also use UCI Opportunity with windowed classification. Opportunity's two label sets, locomotion and high-level activity, plausibly come from separate processes sharing body-worn sensors. For each dataset we report error, label-efficiency curves, and the size of the covariance term relative to the mean-field score.

The decisive real-data check is a correlation across datasets. The benefit over mean field should be largest where the measured posterior cross-chain covariance is largest. If the benefit appears instead on datasets with negligible coupling, it is coming from somewhere other than explaining away, and the mechanism claim fails even if accuracy improves.