Abstract
Probabilistic programming (PP) facilitates complex Bayesian modeling but suffers from difficult-to-debug convergence issues. Traditional diagnostics are reactive and require significant expertise. We present SixthSense, a tool that predicts MCMC convergence by learning from program structure and early runtime signals. By extracting local AST "motifs" and utilizing adaptive mutation for data augmentation, SixthSense achieves over 78% accuracy in static prediction, rising to 83% with early runtime data. It serves as a guided debugging assistant, pinpointing problematic code segments more effectively than standard runtime warnings.
Introduction
Probabilistic programming (PP) allows developers to build complex Bayesian models as computer programs. However, these programs are notoriously difficult to debug. The core challenge is convergence: the inference algorithm (usually Markov Chain Monte Carlo or MCMC) must "settle" into the correct posterior distribution to provide accurate results.
Currently, identifying convergence issues is a reactive process. Developers run a program—which can take minutes or hours—and then check statistical diagnostics like the Gelman-Rubin ($\hat{R}$) statistic. If it hasn't converged, identifying the root cause often requires expert-level statistical knowledge. Existing "big code" machine learning tools struggle here because probabilistic programs are typically small but semantically dense, meaning traditional code-learning approaches miss the subtle statistical relationships that cause failure.
Methods
SixthSense treats convergence prediction as a classification problem. Its architecture consists of several novel components designed to capture the unique semantics of probabilistic code.
Program Motifs and Representation
Instead of looking at long-range paths in the Abstract Syntax Tree (AST), SixthSense extracts "motifs"—short, local fragments of the AST (e.g., a specific distribution call within a loop). These capture the local statistical "flavor" of the code. To handle high dimensionality, Random Discretized Projections (RDP) group similar motifs into buckets, keeping the model size small (<20MB) while preserving structural similarity.
Data Generation via Adaptive Mutation
Because there are relatively few public probabilistic programs available for training, the authors developed a "guided mutation" algorithm. It takes "seed" programs and applies expansive (adding parameters) or reducing (simplifying expressions) transformations to create a diverse dataset of over 10,000 mutants.
Hybrid Feature Set
The model combines static motifs with data features (e.g., skewness or sparsity of input data) and early runtime features (diagnostics from the first 100-200 "warmup" iterations). An interpretable Random Forest classifier allows tracing "non-convergence" predictions back to specific code motifs.
Results
The study utilized 166 seed programs from the Stan repository and over 10,000 generated mutants. SixthSense achieved an average accuracy of 78% using only static code features, outperforming Code2Vec and Code2Seq. When early runtime features were included, accuracy jumped to over 83%.
In a case study of 40 non-converging programs, SixthSense successfully pinpointed the cause of failure in 29 cases, whereas Stan’s built-in runtime warnings only helped in 5 cases. The prediction happens in less than a second, providing a much faster feedback loop than traditional methods.
Conclusion
SixthSense demonstrates that local code structure and early execution signals are highly predictive of long-term convergence in probabilistic programs. By transforming debugging from a reactive statistical analysis into a proactive, guided process, it lowers the barrier to entry for complex Bayesian modeling. Future work includes generalizing across more PPLs and automatically suggesting code fixes.
Summary updated