Multimodal sequence models, such as those that read words, voice, and facial movement together, often learn to rely on one easy stream and ignore the others. This is called modality dominance. The model then breaks when that stream is noisy or missing partway through a clip. Fixing it requires knowing, at each moment, how much each modality is actually contributing. Today that number comes from heuristics: external single-modality "teacher" models, masked forward passes the network was never trained to handle, or costly Shapley-value approximations. Probabilistic circuits can produce the number exactly. These are probability models built from sums and products, arranged so that probabilistic questions are answered exactly in one pass whose cost grows only linearly with model size. In a circuit, removing a modality is a closed-form operation: set its input distributions to one, and the result is the exact prediction without it. The catch is that existing temporal circuits are evaluated one time step after another. That serial dependency makes them slow to train on long, frame-level streams, which are exactly where per-step credibility matters.
This note proposes a fix based on a simple observation. Once the input at a time step is fixed, the step-to-step transition of a temporal circuit is a linear map on the vector of latent-state values, with nonnegative, input-dependent entries. It is the same kind of recurrence that modern linear recurrent networks and state-space models already compute with a parallel scan. A parallel scan is a tree-shaped computation that combines all time steps in logarithmic rather than linear depth. We build a temporal circuit head on top of separate single-modality neural encoders and train the whole stack with this scan. The same scan, rerun with one modality's inputs marginalized out, returns three exact posteriors over the label at every time step: with all modalities, with all but one, and with only one. We then use these exact quantities as a teacher-free reliability signal, both to steer fusion while the stream is running and to allocate gradient to under-trained encoders.
Core Claim
The claim has three parts. Each is testable on its own.
The algebra. In a Dynamic Sum-Product Network template, the outputs from slice $t-1$ feed into slice $t$ through interface nodes. Those interface nodes all share the same scope, namely the past variables. Two structural rules then force the template output to be linear in the incoming interface vector:
- Decomposability means a product node's children never share variables, so at most one child of any product node can depend on the past.
- Smoothness means sum nodes take convex combinations, which are already linear.
Writing $v_t \in \mathbb{R}_{\ge 0}^k$ for the $k$ interface values after step $t$, and $M(x_t)$ for the $k \times k$ nonnegative matrix that the template computes from the step-$t$ inputs $x_t$, we get:
$$v_t = M(x_t)\, v_{t-1}, \qquad P(x_{1:T}) = \mathbf{1}^\top M(x_T)\cdots M(x_1)\, v_0 .$$
Matrix products are associative, so all prefixes can be computed in $O(\log T)$ depth. This is the device behind Resurrecting Recurrent Neural Networks for Long Sequences and BLUR. To avoid underflow, the scan runs in log space, using log-sum-exp in place of addition, which is still associative.
We do not claim the scan trick itself is new. Parallel prefix computation of hidden-Markov-style forward messages is classical algebra. The open question is whether it survives the expressive transitions circuits need, and whether it pays off for multimodal credibility. Neither is addressed in Dynamic Sum Product Networks for Tractable Inference on Sequence Data or Online Structure Learning for Feed-Forward and Recurrent Sum-Product Networks. Both unroll their templates sequentially and handle only single-source streams.
The cost structure. Diagonal transitions make each scan combination cheap. But a diagonal $M$ never mixes latent states, so the circuit collapses into a mixture of independent per-state models. Expressivity therefore sits in the off-diagonal mass, and the cost of the scan is roughly $k^3 \log T$ against $k^2 T$ for sequential unrolling. We will study block-diagonal and diagonal-plus-low-rank transition families that keep the circuit valid while keeping the combination cheap.
The modality-level payoff. Each time step has leaves for each of $M$ modalities. Marginalizing modality $j$ means setting its leaves to one, which yields a modified matrix $M_{\setminus j}(x_t)$. So the exact all-modality, leave-one-out, and single-modality posteriors are $2M+1$ parallel scans over the same sequence. That is a batch dimension, not a serial cost. The per-step credibility is
$$C_{j,t} = D_{\mathrm{KL}}\big(P(Y \mid x_{1:t}) \,\|\, P(Y \mid x_{1:t,\setminus j})\big),$$
which is exactly the quantity Credibility-Aware Multimodal Fusion Using Probabilistic Circuits computes. That paper proves it equals the entropy reduction due to modality $j$, but it computes it only once, on pooled late-fusion summaries from frozen experts.
Research Program
Architecture. Each modality gets its own neural encoder, a diagonal linear recurrent unit. Keeping encoders separate means no shared state can be captured by the dominant stream. Each encoder emits a low-dimensional per-step feature. On top sits a class-conditional temporal circuit:
- a root sum node over labels $Y$;
- per label, a template whose product nodes factor across modalities, with Gaussian leaves over the encoder features;
- sum nodes that mix latent states;
- transition matrices of the block or low-rank structured families above.
Training. The model is trained end to end with the exact conditional log-likelihood $\log P(y \mid x_{1:T})$, plus a small generative term to keep the leaves well defined. Gradients flow through the scan into the encoders.
Steering training with exact quantities. The circuit supplies two exact signals per modality and step. The first is $C_{j,t}$, how much modality $j$ adds beyond the others. The second is the single-modality posterior $P(Y \mid x_{1:t,j})$, what $j$ predicts alone. We combine them into a starvation test.
- A modality whose own posterior is informative but whose $C_{j,t}$ is near zero is redundant. Its information is already carried by the other streams, and it should be left alone. This responds to the "relative balance" finding in BalanceBenchmark that forcing equal contributions hurts.
- A modality whose own posterior is uninformative early in training is under-trained. Its encoder receives an extra gradient budget through an auxiliary loss on that exact single-modality posterior.
This replaces the external, temperature-smoothed teachers of CAT-GS. That paper names teacher-free reliability estimation as open and admits its controller inherits teacher bias. In our design, the "teachers" are coherent marginals of the very model being trained. They cost nothing extra to maintain and cannot disagree with the joint model about what the data say.
Inference on streams. At test time the same per-step credibilities weight fusion while the sequence is running, and missing segments are marginalized exactly rather than imputed. When the full sequence is buffered, a backward scan gives smoothed credibilities. This addresses the causal-versus-bidirectional tension raised by BLUR, without a second neural network.
Positioning against prior work. Mixture-of-Mamba (Mixture-of-Mamba: Enhancing Multi-Modal State-Space Models with Modality-Aware Sparsity) shows that giving each modality its own projections inside a linear recurrence helps multimodal pretraining. It supplies the motivation for modality-specific parameters. It offers no exact marginals and does not measure dominance. BLUR supplies the encoder design and the stability-by-bounded-eigenvalue recipe, but it fuses symmetrically with MLPs. Mitigating Modal Imbalance in Multimodal Reasoning motivates the work by documenting that conflict detection collapses when evidence is split across modalities. It also constrains our claims: its fixes act on attention in large foundation models, a regime we do not target.
Baselines
The naive reference point is plain joint training: separate encoders whose outputs are concatenated and passed to an MLP. BalanceBenchmark reports this vanilla setup at 65.50% accuracy on CREMA-D, and CAT-GS reports standard joint training at 67.47%. We rerun it with our encoders so that differences reflect fusion and training, not backbone.
The strongest existing method on dominance is CAT-GS. It reports 86.29% accuracy on CREMA-D, above the 85.89% for G²D, and 67.55% on the UR-FUNNY audio-video-text task, against 65.23% for full C-MFN in UR-FUNNY: A Multimodal Language Dataset for Understanding Humor. For CMU-MOSEI, the reference is Graph-MFN's 76.9% binary sentiment accuracy from Multimodal Language Analysis in the Wild: CMU-MOSEI Dataset and Interpretable Dynamic Fusion Graph. For fair comparison, we run CAT-GS on top of our identical linear-recurrent encoders, train its unimodal teachers on the same data, and give it the same hyperparameter search budget and GPU-hours. "Beating" it means two things. First, matching its clean accuracy within about one point. Second, clearly exceeding it when a modality is corrupted or dropped in random mid-sequence windows, while using no external teacher.
The strongest neural sequence-fusion competitor is a Mixture-of-Mamba-style block with modality-specific projections, trained with modality dropout. It is run with the same parameter count, the same dropout schedule, and the same budget.
The baseline for the scan contribution is the sequentially unrolled circuit of Dynamic Sum Product Networks for Tractable Inference on Sequence Data. That paper reports 2.9 seconds for inference on ArabicDigits, against roughly 465 seconds for a Dynamic Bayesian Network. We implement it with our identical template on the same GPU. "Beating" it means:
- log-likelihoods identical to numerical precision;
- wall-clock training speedup that grows with $T$;
- a crossover sequence length, which we report honestly.
For the credibility ingredient, the late-fusion Direct-PC and credibility-weighted mean of Credibility-Aware Multimodal Fusion Using Probabilistic Circuits are the reference. That paper reports 96.48% AUROC on AV-MNIST and 57.97% accuracy on SUN RGB-D, versus 54.55% for an MLP.
Two ablations isolate the key ingredients. The first keeps our circuit head but pools features before fusion, so credibility is computed once per sequence instead of per step. The second keeps the per-step credibilities but drops the starvation-driven gradient budget.
Why not just…?
Why not just train a neural linear-recurrent fusion model with modality dropout and get leave-one-out predictions by masking inputs? This is the most serious threat. It is cheap, and modality dropout does teach a network to predict with a modality removed. The difference is that masked predictions are approximations whose quality depends on the dropout schedule. They also carry no guarantee that the full, leave-one-out, and single-modality predictions are mutually consistent, so the KL "credibility" between them can reflect miscalibration rather than information. The synthetic-credibility experiment below measures this directly. If dropout-masked credibilities track the ground-truth informativeness as well as the exact ones, and drive gating equally well, then exactness buys nothing. We would then report that and retreat to the scan contribution alone.
Why not just add a unimodal auxiliary classifier per encoder, as in gradient blending, or use CAT-GS's teachers? Separate heads are trained independently of the fused model. They can disagree with it about what each modality implies, and CAT-GS itself reports that biased teachers propagate into the gating. Our single-modality posteriors are marginals of the same joint distribution. The ablation that swaps them for independently trained heads, with everything else held fixed, tells us whether coherence matters for balance. Encoder linear-probe accuracy per modality after joint training is the starvation diagnostic.
Why not just run the existing temporal circuit sequentially, since the sequences are short? On word-aligned CMU-MOSEI and UR-FUNNY, sequences are tens of steps, and the scan gives little speedup. We concede this. The scan matters for frame-level streams: CREMA-D audio-video at frame rate, ArabicDigits-style speech, and long sensor fusion. There, per-step credibility is most useful and sequential unrolling, together with backpropagation through it, dominates training time. The speed experiments are run where $T$ reaches the thousands, and the accuracy claims do not depend on them.
Proposed Experiments
Correctness and speed. We train identical circuits by scan and by sequential unrolling on ArabicDigits, PenDigits, and synthetic streams with $T$ from 64 to 4096. We verify equal log-likelihoods. We then plot wall-clock time and memory against $T$ and against the interface width $k$, for the dense, block, and low-rank transition families. We also report the expressivity cost of each structured family as held-out log-likelihood.
Credibility fidelity. This is the decisive check for exactness. We build temporally corrupted AV-MNIST sequences and CREMA-D clips where noise of known strength is injected into one modality during random windows, so the true per-step informativeness is known. We then compare how well three estimates of $C_{j,t}$ track it, measured by rank correlation and by window-detection AUROC:
- exact $C_{j,t}$ from our circuit;
- dropout-masked credibilities from the Mixture-of-Mamba baseline;
- CAT-GS teacher margins.
Balance and robustness. On CREMA-D, UR-FUNNY, and CMU-MOSEI we report three things. The first is clean accuracy against the numbers above. The second is accuracy under mid-sequence modality dropout and corruption at increasing rates. The third is per-modality encoder probe accuracy after joint training. The proposal succeeds if it matches CAT-GS on clean data, outperforms it under mid-sequence degradation, and leaves weaker modalities' encoders measurably less starved. The single result that would most change our minds is if dropout-masked neural credibilities match exact ones in the fidelity test. In that case the probabilistic machinery is not earning its keep for dominance.
References
- Resurrecting Recurrent Neural Networks for Long Sequences
- Bidirectional linear recurrent models for sequence-level multisource fusion
- Dynamic Sum Product Networks for Tractable Inference on Sequence Data
- Online Structure Learning for Feed-Forward and Recurrent Sum-Product Networks
- Credibility-Aware Multimodal Fusion Using Probabilistic Circuits
- BalanceBenchmark: A Survey for Multimodal Imbalance Learning
- CAT-GS: Balanced Multimodal Learning via Calibrated Gating and Fusion Surgery
- Mixture-of-Mamba: Enhancing Multi-Modal State-Space Models with Modality-Aware Sparsity
- Mitigating Modal Imbalance in Multimodal Reasoning
- UR-FUNNY: A Multimodal Language Dataset for Understanding Humor
- Multimodal Language Analysis in the Wild: CMU-MOSEI Dataset and Interpretable Dynamic Fusion Graph