Abstract
This paper introduces Probability Engineering, a novel paradigm that shifts the focus of deep learning from traditional statistical distribution fitting to the active modification of probability distributions as "engineering artifacts." By treating distributions as malleable components, this framework addresses critical bottlenecks in modern AI, including Bayesian sampling inefficiency, data heterogeneity in Federated Learning, and factual hallucinations in generative models. Through a series of novel algorithms—including SPOS, Fed-CBS, and ARTIST—this work demonstrates how systematic distribution refactoring can achieve superior performance in high-dimensional, resource-constrained, and complex generative tasks.
Introduction
While deep learning has achieved remarkable success, its reliance on classical probabilistic modeling often hits a wall when faced with the "messy" reality of modern AI. Traditional methods prioritize mathematical purity, attempting to perfectly fit theoretical distributions. However, this approach frequently fails in high-dimensional parameter spaces, under strict hardware constraints, or when dealing with dynamic user data.
Probability Engineering bridges the gap between elegant theory and practical necessity. Instead of asking "What is the true distribution?", this paradigm asks "How can we reshape this distribution to make it faster, safer, or more accurate?" This dissertation applies this mindset to three core domains: Bayesian sampling, Edge AI, and Generative AI.
Methodological Framework
The core of the Probability Engineering process involves identifying a target distribution, disentangling its affected components, and refactoring it to meet specific operational goals. Key algorithmic contributions include:
1. Stochastic Particle-Optimization Sampling (SPOS)
To address "particle collapse" in traditional Bayesian methods like SVGD, SPOS injects stochastic noise into particle updates. This bridges the gap between optimization and MCMC sampling, ensuring diverse and representative samples in complex probability landscapes.
2. Federated Class-balanced Sampling (Fed-CBS)
In Edge AI, Federated Learning (FL) often struggles with non-IID (heterogeneous) data. Fed-CBS utilizes Fully Homomorphic Encryption (FHE) to calculate class imbalances across decentralized clients without compromising privacy. It then "engineers" client selection probabilities to ensure each training round utilizes a balanced global batch, significantly accelerating convergence.
3. Self Logits Evolution Decoding (SLED)
To combat hallucinations in Large Language Models (LLMs), SLED treats the output as an evolving process. It contrasts final-layer predictions with early-layer "latent knowledge," effectively using the model's internal representations as a fact-check filter to steer completions toward truthfulness.
4. ARTIST: Disentangled Diffusion
For text-to-image generation, the ARTIST framework separates the generation of text structure (the "what" and "where") from visual style. By using a dedicated module for text masks before applying diffusion, the model achieves high-fidelity, legible text rendering within images.
Experimental Validation and Results
The proposed methods were validated across diverse benchmarks, ranging from UCI public datasets to specialized LLM and text-to-image evaluations.
- Efficiency in Edge AI: Fed-CBS reached target accuracy in 384 communication rounds, compared to 537–1080 rounds for standard baselines in highly heterogeneous settings.
- Factuality in LLMs: SLED improved factual accuracy by 5–13% across LLaMA and Gemma models on benchmarks like TruthfulQA and FACTOR, without requiring additional training or external data.
- Legibility in Image Generation: On the MARIO-Eval benchmark, ARTIST achieved an OCR accuracy of 0.7373, significantly outperforming the previous state-of-the-art (0.5712).
- Knowledge Distillation: ReAugKD (Retrieval-Augmented Knowledge Distillation) achieved state-of-the-art results on 5 out of 6 GLUE tasks with less than 3% latency overhead.
Discussion and Key Insights
The shift toward Probability Engineering yields several profound insights for the AI community:
- The "Early Layer" Truth: LLMs often possess correct factual information in their middle layers, but this signal is frequently diluted by the final output layer. Contrasting these layers provides a powerful mechanism for factuality.
- Disentanglement for Coherence: The "gibberish text" problem in diffusion models stems from trying to learn pixels and semantic letters simultaneously. Separating these concerns into dedicated modules is essential for legibility.
- Privacy-Preserving Optimization: Fed-CBS demonstrates that we can solve the non-IID data problem in FL without sacrificing privacy by using encrypted inner products to compute label distribution similarities.
Conclusion and Future Work
Probability Engineering provides a pragmatic path forward for AI development, allowing researchers to embed ethical, hardware, and accuracy constraints directly into the probabilistic framework. Future research will explore applying these principles to AI agents to maintain explicit "belief states" in uncertain environments and developing automated systems to identify which distribution components require refactoring to meet new constraints.
Summary updated