Abstract
Various problems in Engineering and Statistics require the computation of the likelihood ratio function of two probability densities. In classical approaches the two densities are assumed known or to belong to some known parametric family. In a data-driven version we replace this requirement with the availability of data sampled from the densities of interest. For most well known problems in Detection and Hypothesis testing we develop solutions by providing neural network based estimates of the likelihood ratio or its transformations. This task necessitates the definition of proper optimizations which can be used for the training of the network. The main purpose of this work is to offer a simple and unified methodology for defining such optimization problems with guarantees that the solution is indeed the desired function. Our results are extended to cover estimates for likelihood ratios of conditional densities and estimates for statistics encountered in local approaches.
Introduction
In engineering and statistics, the likelihood ratio (LR) is the fundamental tool for decision-making. Whether determining if a signal is present or if a process has changed, the standard approach involves comparing the ratio of two probability densities against a threshold. Traditionally, these densities are assumed to be known or to belong to a simple parametric family, such as a Gaussian distribution. However, for complex data like images or video, finding a parametric model is often impossible. While non-parametric methods like kernel density estimation exist, they are difficult to tune and require estimating two separate densities, which compounds errors.
This paper addresses the challenge of direct estimation: training a neural network to output the likelihood ratio (or its log) using only samples from two distributions, bypassing the need to explicitly model the underlying density functions.
Methodology: A Unified Optimization Framework
The core contribution is a unified optimization framework. The authors define a general cost function: $$J(u) = E_0[\phi(u(X)) + r(X)\psi(u(X))]$$ where $r(X)$ is the unknown likelihood ratio. By carefully choosing the scalar functions $\phi$ and $\psi$, the authors prove that the global minimizer of this cost is exactly the desired transformation of the likelihood ratio.
Key Innovations
- Data-Driven Reformulation: The cost function is rewritten to utilize samples from two different datasets ($X_0$ and $X_1$), effectively eliminating the unknown $r(X)$ from the training objective.
- Exponential Loss for Log-LR: The authors propose a specific loss function for estimating the log-likelihood ratio. This approach naturally handles the non-negativity constraint of likelihood ratios, as $e^u$ is always positive.
- Time Series Extension: The framework extends to estimate conditional likelihood ratios for Markovian processes, which is critical for real-time applications like the CUSUM (Cumulative Sum) algorithm for change detection.
- Local Statistics: A method is derived to estimate "local statistics" used to detect subtle changes in a distribution's parameters.
Experimental Evaluation
The proposed methods were evaluated using both synthetic and real-world datasets:
- Synthetic Gaussian Data: High-dimensional (k=10) Gaussian vectors were used to compare neural network performance against the theoretical optimum where densities are perfectly known.
- MNIST Database: The "4" vs. "9" handwritten digit classification task served as a benchmark for high-dimensional, non-parametric data where traditional models often struggle.
Key Results
- Block Testing: In hypothesis testing with blocks of 20 samples, the proposed Log-LR estimator (using exponential loss) nearly matched the theoretical optimum. Standard Mean Square Error approaches showed clearly inferior performance.
- Classification Accuracy: On the MNIST task, the proposed likelihood-based losses achieved error rates comparable to state-of-the-art Hinge and Cross-Entropy losses.
- Sequential Detection: In change detection simulations, the neural network-based CUSUM approached the performance of the "Exact CUSUM" as the number of training samples increased.
Discussion and Limitations
A significant insight from this work is that estimating the log-likelihood ratio is superior to estimating the raw ratio. Raw LR estimation often struggles with non-negativity constraints and performs poorly in block testing. Furthermore, neural networks proved robust to deterministic features in data (e.g., zero-valued pixels in MNIST) that typically cause parametric models to fail.
However, challenges remain. While the functional optimization problem is convex, the actual training of a neural network is not, due to its non-linear structure. This can lead to local minima during gradient descent, as illustrated in the figure above. Additionally, sequential detection requires large training sets to capture "rare events" in the distribution tails; performance degrades significantly if the training set is insufficient. Future work will focus on determining the sample complexity required to guarantee specific performance levels in sequential hypothesis testing.
Summary updated