Abstract
Maximum Entropy (MaxEnt) is a standard technique for density estimation from presence-only data. However, real-world datasets are frequently incomplete due to sensor failures or human error. Traditional approaches either discard incomplete samples or rely on error-prone imputation methods. This paper proposes a robust extension of MaxEnt that handles missing feature values by redefining optimization constraints to apply only to observed data. Our approach avoids the biases of imputation, remains computationally efficient, and demonstrates superior performance across various missing-data regimes, including Missing Completely at Random (MCAR), Missing at Random (MAR), and Not Missing at Random (NMAR).
Introduction
In many domains, such as ecological niche modeling or industrial fault diagnosis, researchers must estimate probability densities from "presence-only" data—observations of where an event occurred without corresponding "absence" data. Maximum Entropy (MaxEnt) is the preferred tool for these tasks. However, standard MaxEnt implementations require complete feature vectors, a requirement rarely met in practice.
Common workarounds involve discarding incomplete records or performing data imputation (e.g., filling missing values with the mean). Discarding data wastes information, while imputation can introduce significant bias, as the model effectively "learns" from its own guesses. This paper introduces a version of MaxEnt that learns directly from available data without imputation.
Methodology: Redefining Constraints
The core innovation is the modification of the MaxEnt objective's constraints. In standard MaxEnt, the model is constrained to match the global empirical mean of each feature. The authors introduce a missingness indicator $o_j(x)$, which is 1 if feature $j$ is observed and 0 otherwise.
Instead of a global mean, the model is constrained such that the conditional expectation—the mean of the feature given that it was observed—matches the empirical data. This ensures the model only relies on verified information.
Optimization and Efficiency
The resulting optimization problem remains convex. The authors provide a greedy coordinate descent algorithm using Newton’s method to find optimal parameters ($\lambda$). This approach is computationally scalable, making it suitable for large-scale real-world applications.
Experimental Evaluation
The method was evaluated using:
- Synthetic Data: 5,000 datasets generated to test performance under MCAR, MAR, and NMAR regimes.
- UCI Datasets: Six real-world datasets (e.g.,
horse-colic,house-votes) with naturally occurring missing values.
Results and Discussion
Performance across Missingness Regimes
On synthetic data, the proposed "Missing" method consistently achieved higher out-of-sample log-likelihoods than mean or EM-based imputation. Crucially, the method proved robust even in Not Missing At Random (NMAR) scenarios, where the missingness itself is correlated with the underlying values—a situation where traditional imputation often fails.
Robustness on Real-World Data
In tests on UCI datasets, the proposed method was always competitive with the best imputation strategies and was never significantly outperformed. This suggests that the theoretical advantages of avoiding the "imputation trap" translate well to practical, noisy datasets.
Conclusion
By redefining the constraints of the Maximum Entropy objective, this work provides a theoretically sound and practically efficient way to handle incomplete presence-only data. The method avoids the biases of imputation and the information loss of data deletion, offering a robust tool for density estimation in real-world scenarios where data is often imperfect.
Summary updated