Introduction
In complex multi-modal environments—such as a conference room equipped with multiple cameras and microphones—the fundamental challenge of data association arises: which audio stream corresponds to which visual subject? Traditional solutions often employ Windowed Factorization Tests (WFT), which use "sliding windows" to detect instantaneous correlations between signals. However, these methods are inherently "forgetful," failing to learn long-term subject characteristics, and they only detect the presence of a connection while ignoring the specific parameters (e.g., a person's unique appearance) that define it.
This paper introduces the Hidden Factorization Markov Model (HFactMM), a framework designed to learn both the evolving dependency structure and the unique features of data streams simultaneously in an unsupervised manner.
The HFactMM Framework
The core innovation of HFactMM is treating the dependency structure itself as a hidden state. In a standard Hidden Markov Model (HMM), states typically represent labels. In HFactMM, each hidden state $S_t$ represents a specific graphical factorization of the observed variables.
As shown in Figure 1, the model partitions $N$ random variables into independent subsets. For instance, one state might represent a scenario where Audio 1 and Video A are dependent, while Video B is independent. The model uses an Expectation-Maximization (EM) algorithm for learning, with a modified M-step that updates parameters based on the specific factorization of the current state. Inference is performed via Viterbi decoding to determine the most likely sequence of associations (e.g., "who was speaking when") across the entire stream.
Key Insights and Results
The primary advantage of HFactMM lies in its ability to exploit both structural and parametric information. While sliding window methods like WFT only care if signals correlate in the moment, HFactMM learns the "appearance" or "sound" of subjects over the entire duration. If a subject has a distinct appearance, the model uses that information to maintain association even when the instantaneous correlation signal is weak.
Experimental results on the CUAVE corpus and custom datasets demonstrate this superiority. On the CUAVE dataset, HFactMM achieved 88.11% accuracy, outperforming supervised state-of-the-art methods. In scenarios with significant appearance changes (like gaze or pose shifts), HFactMM remained robust where WFT failed. As illustrated in Figure 2, as the parametric distinctness between states increases (represented by distance $D$ and distribution shape $\rho^2$), the error probability for HFactMM drops significantly faster than for the "forgetful" WFT baseline.
Conclusion
HFactMM provides a powerful, unsupervised approach to multi-modal data association by integrating dynamic structure learning with parametric modeling. While the model currently assumes all interaction states are visited during a sequence and faces potential complexity challenges with many simultaneous speakers, it represents a significant step toward robust, "on-the-fly" subject identification in unconstrained environments. Future work may involve refining transition priors to handle noisier data and scaling the state space for more complex multi-speaker scenarios.
Summary updated