The Poisson Compound Decision Problem Revisited

Lawrence D. Brown, Eitan Greenshtein, Ya’acov Ritov

TL;DR

This paper revisits the classic Poisson compound decision problem, proposing novel smoothing adjustments to the existing "classical" empirical Bayes estimator. These adjustments, along with an alternative approach transforming the problem to a normal setting, significantly improve estimation performance, especially in challenging scenarios. The authors also introduce a non-standard cross-validation method for selecting a crucial smoothing parameter.

What Problem Does This Paper Address?

The paper tackles the problem of estimating a vector of unknown Poisson means, $\lambda = (\lambda_1, \dots, \lambda_n)$, when we observe $n$ independent Poisson random variables $Y_i \sim \text{Po}(\lambda_i)$. This is known as the Poisson compound decision problem. While a well-established "classical" empirical Bayes solution exists, the authors argue that it often requires smoothing for better performance, particularly in high-dimensional settings where $n$ is large. The core challenge is to improve upon this classical estimator, which can be slow to converge and exhibit suboptimal performance even with a substantial number of observations. They aim to provide more robust and accurate estimators by introducing smoothing techniques and exploring alternative problem formulations.

Methods

The authors propose a three-step adjustment to the classical Robbins estimator:

  1. Step 1: Smoothing via Corrupted Observations: They introduce a "corrupted" observation $Z_i = Y_i + N_i$, where $N_i \sim \text{Po}(h)$ is independent Poisson noise with parameter $h$. This transforms the problem into estimating means for these corrupted observations. The resulting suboptimal decision function, $\delta_{h,1}(z)$, is derived and then estimated using a Robbins-type procedure. The parameter $h$ acts as a "smoothing parameter."
  2. Step 2: Rao-Blackwellization: The estimator from Step 1 is further improved by Rao-Blackwellization, which involves taking the conditional expectation of $\delta_{h,1}(Z_i)$ with respect to the added noise $N_i$. This leads to a new estimator, $\delta_{h,2}(y)$.
  3. Step 3: Isotonic Regression (Monotonization): Since the previous steps might not yield a monotone decision function (which is desirable for Poisson models due to the monotone likelihood ratio property), they impose monotonicity using isotonic regression via the pooling adjacent violators algorithm. This results in the final adjusted estimator, $\Delta_h = \delta_{h,3}$.

Additionally, they explore an alternative approach that transforms the Poisson problem into a normal compound decision problem using a variance-stabilizing transformation (e.g., $Z_i = 2\sqrt{Y_i} + q$). This allows them to apply established methods for the normal case and then transform the results back.

A key contribution is a non-standard cross-validation method for selecting the smoothing parameter $h$. This method involves splitting the data into two sub-samples: one for constructing estimators and another for validation, leveraging the properties of Poisson and binomial distributions.

Datasets

The paper uses simulated data for its extensive simulation studies. They also present a real-data example involving car accident data from 109 towns in Israel in July 2008. This real-data example is used to illustrate the performance of their proposed methods in a practical setting.

Key Insights and Ideas

  • The "classical" estimator needs smoothing: Even with a large number of observations, the standard empirical Bayes estimator for the Poisson problem can be significantly improved by introducing smoothing. This is counter-intuitive to the idea that more data should directly lead to better estimates without further adjustments.
  • Transforming to normality is effective but has limitations: The approach of transforming the Poisson problem to a normal one is a viable strategy and often yields comparable results. However, it can be less stable and perform less well in certain configurations, particularly with tightly clumped or small $\lambda$ values, or when tail approximations are critical.
  • Monotonicity is crucial: Imposing monotonicity on the estimator, even after smoothing, provides substantial improvements, highlighting the importance of leveraging the underlying distributional properties.
  • A novel cross-validation for smoothing parameter selection: The proposed cross-validation method offers a data-driven way to choose the smoothing parameter $h$, which is critical for the performance of the adjusted estimators. This is a significant practical contribution.
  • Asymptotics can be misleading for practical sample sizes: While theoretical asymptotics suggest that Robbins' estimator is efficient in certain sparse settings, simulations show that the "kick-in" point for these asymptotics can be at very large sample sizes, making adjusted estimators more beneficial in practice.

Key Results

The paper presents extensive simulation results comparing their proposed adjusted Poisson estimators ($\Delta_h$) and the transformed normal estimators ($\Delta_{N,h}$) against the classical Robbins estimator ($\delta_{h,2}$ with $h=0$) and its monotonized version ($\Delta_0$).

  • Significant Risk Reduction: In various scenarios, the adjusted Poisson estimators ($\Delta_h$) achieve substantial reductions in risk compared to the classical Robbins estimator. For instance, in Example 1, the average loss for the classical procedure was 4335, while the adjusted procedure with smoothing and monotonicity reduced it to 30.
  • Stability of Adjusted Poisson Estimators: The adjusted Poisson estimators demonstrate remarkable stability with respect to the choice of the smoothing parameter $h$, as seen in the tables. This is an advantage over the transformed normal method, which can be more sensitive to $h$.
  • Real Data Performance: In the real data example (Table 6), the classical Poisson-EB adjusted for monotonicity ($h=0$) performed best, but the adjusted Poisson estimators with $h>0$ were only slightly inferior and showed good stability. The modified normal estimators performed worse.
  • Cross-validation effectiveness: The cross-validation method for selecting $h$ showed impressive performance, often yielding risks close to or even better than the best individual smoothing parameter choices across experiments.

Limitations and Future Work

  • Asymptotic "Kick-in" Point: The authors note that the asymptotic results for Robbins' estimator, while theoretically sound, may only become relevant for very large sample sizes, potentially misleading practitioners for moderately large $n$.
  • Limitations of Normal Approximation: The transformed normal approach, while useful, has limitations in accurately approximating the tails of the Poisson distribution, which can affect performance in certain configurations.
  • Future Work: The authors suggest that their methods can be modified and specialized for other loss functions and generalized to include additional covariates. They also express interest in studying these extensions in the future. The paper also mentions that the NPMLE (nonparametric maximum-likelihood estimator) is an alternative that automatically yields a monotone and smooth decision function, and its asymptotic properties are a topic for future research.

Summary updated

Abstract

The compound decision problem for a vector of independent Poisson random variables with possibly different means has a half-century-old solution. However, it appears that the classical solution needs smoothing adjustment. We discuss three such adjustments. We also present another approach that first transforms the problem into the normal compound decision problem. A simulation study shows the effectiveness of the procedures in improving the performance over that of the classical procedure. A real data example is also provided. The procedures depend on a smoothness parameter that can be selected using a nonstandard cross-validation step, which is of independent interest. Finally, we mention some asymptotic results.
BibTeX
@article{brown2013poisson,
  author = {Lawrence D. Brown and Eitan Greenshtein and Ya’acov Ritov},
  title = {{The Poisson Compound Decision Problem Revisited}},
  journal = {Journal of the American Statistical Association},
  year = {2013},
  doi = {10.1080/01621459.2013.771582},
}
Figure