Introduction
The analysis of longitudinal count compositional data is critical in fields such as microbiology and natural language processing. The Multinomial Logistic-Normal Dynamic Linear Model (MLN-DLM) is considered the gold standard for such data due to its ability to capture complex correlations and time-varying trends. However, traditional Bayesian inference methods, such as those implemented in Stan, face significant computational bottlenecks and numerical instabilities when scaled to high-dimensional datasets.
The Fenrir Framework
We introduce Fenrir, a high-performance C++ library designed to address these challenges. Fenrir achieves inference speeds up to 1,000 times faster than current state-of-the-art tools by integrating several key innovations:
- Gradient Filtering for MAP Estimation: Fenrir utilizes a recursive filtering approach, analogous to a Kalman filter, to compute gradients sequentially. This avoids the numerical instability associated with pre-computing large covariance matrices for entire time series.
- Debiased Multinomial Dirichlet Bootstrap (DMDB): To quantify uncertainty without the overhead of Markov Chain Monte Carlo (MCMC) sampling, Fenrir employs a novel bootstrap method. It uses the Maximum A Posteriori (MAP) estimate to generate pseudo-observations, approximating the posterior distribution efficiently.
- Optimized Implementation: The core engine is written in C++ using the Eigen and Boost libraries, provided with an accessible R interface.
Experimental Evaluation
Fenrir was evaluated using both synthetic and real-world datasets. Synthetic tests varied the number of multinomial categories (25 to 100) and timepoints (200 to 4,000) to assess scalability.
As shown in the evaluation across timepoints, Fenrir maintains a near-constant total time compared to the exponential growth seen in Stan. Furthermore, the ratio of log probabilities demonstrates rapid convergence across different temporal scales.
Scaling with respect to the number of categories further highlights Fenrir's efficiency. While Stan's total time increases significantly with dimensionality, Fenrir remains highly performant, enabling the analysis of datasets with over 100 categories—a task previously considered computationally unreachable.
Key Results and Discussion
In comparative simulations, Fenrir produced effective samples nearly 800 to 1,000 times faster than optimized Stan implementations. For instance, Fenrir generated 2,000 samples in 0.16 seconds, whereas Stan required 130 seconds. This efficiency stems from the finding that the MAP estimate serves as an exceptionally accurate proxy for the posterior mean in MLN-DLMs. Despite the massive speedup, Fenrir maintains high accuracy; posterior means and 95% credible intervals were virtually identical to those produced by Stan.
On real-world artificial gut microbiome data, Fenrir successfully identified sub-daily oscillations in Enterobacteriaceae, reproducing complex biological findings in a fraction of the time.
Conclusion and Future Work
Fenrir represents a significant leap in the scalability of Bayesian MLN-DLMs. While the DMDB approximation may face challenges with extremely sparse data, its performance on high-dimensional count data makes it a powerful tool for modern genomics. Future work will explore the integration of Fenrir as a proposal distribution for Sequential Monte Carlo methods to further enhance accuracy in sparse data regimes.
Summary updated