Abstract
The accurate measurement of security metrics is a critical research problem because an improper or inaccurate measurement process can ruin the usefulness of the metrics, no matter how well they are defined. This is a highly challenging problem particularly when the ground truth is unknown or noisy. In contrast to the well perceived importance of defining security metrics, the measurement of security metrics has been little understood in the literature. In this paper, we measure five malware detection metrics in the absence of ground truth, which is a realistic setting that imposes many technical challenges. The ultimate goal is to develop principled, automated methods for measuring these metrics at the maximum accuracy possible. The problem naturally calls for investigations into statistical estimators by casting the measurement problem as a statistical estimation problem. We propose statistical estimators for these five malware detection metrics. By investigating the statistical properties of these estimators, we are able to characterize when the estimators are accurate, and what adjustments can be made to improve them under what circumstances. We use synthetic data with known ground truth to validate these statistical estimators. Then, we employ these estimators to measure five metrics with respect to a large dataset collected from VirusTotal. We believe our study touches upon a vital problem that has not been paid due attention and will inspire many future investigations.
Introduction
In the cybersecurity domain, evaluating the efficacy of malware detectors typically relies on "ground truth"—a definitive label of whether a file is malicious or benign. However, obtaining this truth is a significant hurdle. Manual expert analysis is unscalable, and third-party blacklists are often incomplete or lagging. Consequently, researchers frequently resort to "majority voting" among multiple antivirus engines as a proxy for truth. This paper argues that such "pretended ground truth" is fundamentally flawed and proposes a rigorous statistical framework to estimate detection metrics even when the true status of every file remains hidden.
The Fallacy of Majority Voting
The reliance on majority voting assumes that the collective opinion of multiple imperfect detectors will converge on the truth. However, if the detectors share common biases or have mediocre individual performance, the majority vote becomes a source of systematic error.
As shown in Figure 1, the probability of a correct majority vote ($p_{11}$) remains high only when individual error rates ($p_-$) are low. Once error rates exceed a certain threshold (approximately 0.5), the accuracy of the majority vote collapses regardless of the number of detectors. This demonstrates that a defender needs a small set of "very good" detectors (error rates < 0.1) to achieve reliable consensus; otherwise, the majority vote significantly underestimates metrics like false positive rates.
Methodology: From Naïve to Adjusted Estimators
To address this, the researchers framed the measurement as a statistical estimation problem using a three-step approach:
- Naïve Estimators: Initial metrics are calculated based on the simple majority vote (the "voted label").
- Theoretical Bias Analysis: Using the Central Limit Theorem, the authors mathematically proved that these naïve estimators are inherently biased, showing exactly how far the majority vote deviates from reality.
- Adjusted Estimators: By applying the Method of Moments, they developed "adjusted" formulas designed to subtract the predicted bias. These estimators account for the heterogeneous error rates across different detectors.
Empirical Validation and Results
The framework was validated using synthetic datasets (50,000–100,000 files) where ground truth was known, allowing for a direct comparison between naïve and adjusted methods.
As illustrated in Figure 2, the adjusted estimators (bottom row) consistently exhibit lower bias than their naïve counterparts (top row), particularly when the number of detectors is small or their quality is only "fair." This confirms that the statistical adjustments successfully mitigate the errors introduced by relying on imperfect consensus.
When applied to a real-world VirusTotal dataset of over 9.4 million files and 47 detectors, the model estimated the true prevalence of malware ($\pi_1$) to be approximately 58.6%. The analysis further revealed that none of the 47 engines qualified as "very good" across the entire dataset, with most vendors trading off low false-positive rates for high false-negative rates.
Conclusion and Future Work
This research provides a principled way to evaluate security tools in the absence of ground truth. While the current model assumes detector independence and homogeneous error rates across file types, it lays the groundwork for more sophisticated models. Future investigations will focus on accounting for detector correlation (e.g., shared OEM engines) and implementing weighted voting schemes to further refine the accuracy of these statistical estimates.
Summary updated