Imagine trying to understand global maritime traffic by analyzing only the shipwrecks that make the evening news. You would likely conclude that the oceans are exclusively populated by massive oil tankers caught in catastrophic, once-in-a-century storms.
This is the core dilemma facing empirical research that relies on public or administrative records. Whether analyzing software vulnerabilities, criminal justice patterns, or corporate data breaches, researchers frequently treat public datasets as neutral reflections of reality. In practice, however, data is the output of a highly selective recording process. When we train machine learning models or draw statistical conclusions from these records, we are often modeling the behavior of the gatekeepers—reporters, corporate PR teams, legal clerks, and police officers—rather than the underlying phenomena themselves.
The Filtering Mechanisms: Why Data Goes Missing
To understand why public records are skewed, we must examine the specific incentives and constraints that dictate whether a real-world event transitions into a digital data point. Across domains, this filtering happens through several distinct mechanisms.
1. Reputational Risk and Strategic Secrecy
In cybersecurity and corporate governance, an incident is often hidden unless disclosure is legally compelled. As Sarabi et al. point out, public breach datasets are heavily skewed toward highly visible or legally mandated disclosures. This leaves stealthy, non-regulated threats like corporate espionage completely unrepresented in risk models.
This institutional silence extends to broader cyber threat forecasting. Almahmoud et al. note that victims frequently suppress incident details to avoid reputational damage, obscuring the true scale and frequency of attacks. Even when examining the cybercrime economy directly, the data remains uneven; Oosthoek et al. observe that the inherent opacity of ransomware operations means certain ransomware families are disproportionately represented in research simply because their payment data happens to be publicly accessible.
2. The "Silent Patch" in Software Security
We often treat public vulnerability registries, like the Common Vulnerabilities and Exposures (CVE) database, as exhaustive catalogs of software risk. However, Zheng et al. demonstrate that empirical security research suffers from severe selection bias because it relies entirely on these documented entries, completely missing bugs that are quietly resolved by developers without a public advisory.
Sun et al. echo this concern, showing that public databases systematically miss zero-day exploits and flaws resolved via silent patches. When analyzing specific software ecosystems, such as remote learning platforms, Cohney et al. emphasize that relying strictly on CVEs yields an incomplete picture because many internal security flaws remain unaddressed or unrecorded entirely.
3. Institutional Discretion and the "Dark Figure" of Crime
In criminal justice, the gap between actual events and official administrative records is known as the "dark figure of crime." Fogliato et al. explain that incident-based crime datasets omit a massive portion of actual victimizations because they only capture offenses that are officially reported to law enforcement and not subsequently dismissed by police.
This reporting threshold is highly dependent on human behavior and institutional choices. For instance, Karystianis et al. note that surveillance systems for domestic violence are fundamentally constrained by an individual's willingness to report an incident and the subjective recording choices of responding officers, creating a significant chasm between documented data and actual prevalence.
4. Newsworthiness and Political Suppression
When datasets are scraped from news media or public incident trackers, they inherit the commercial and political biases of journalism. The author of the AI Incident Database, McGregor, notes that incident repositories relying on public media are naturally limited by the visibility and newsworthiness of specific failures, leaving internal or quiet corporate harms undocumented.
In more sensitive contexts, this visibility is further distorted by active institutional intervention. Zhang et al. show that public media-derived datasets on sensitive topics like suicide suffer from institutional suppression and geographic disparities in media coverage, heavily underestimating the true incidence in rural or less-monitored regions.
5. The Selective Curation of Legal Records
Even within formal legal systems, the public record is a curated subset of reality. Östling et al. point out that legal AI datasets built on published court records are heavily skewed toward landmark or complex cases, limiting an AI's ability to learn the nuances of routine, everyday judicial decision-making.
Furthermore, Lo et al. find that cases with specific outcomes—such as those where the plaintiff prevails—generate significantly more public documentation, creating an unrepresentative sample of case outcomes. This is reinforced by Xue et al., who observe that a vast portion of court verdicts are simply withheld from public release, distorting our view of the broader judicial landscape.
The Downstream Danger: Why Models Break
Why does this selection bias matter so much? If data were missing completely at random, it would merely reduce statistical power. But public reporting bias is systematically non-random. When a machine learning model is trained on biased administrative data, it doesn't learn the underlying phenomenon; it learns the selection mechanism.
[ Real-World Events ]
│
▼ (Filter 1: Willingness to report / discover)
[ Reported Incidents ]
│
▼ (Filter 2: Institutional discretion / silent fixes)
[ Official Records ]
│
▼ (Filter 3: Public disclosure mandates / newsworthiness)
[ Publicly Available Datasets ] ──► Trained Model ──► Distorted Predictions
Consider predictive policing. A model trained on arrest records does not learn where crime occurs; it learns where police make arrests. As Fogliato et al. demonstrate, when researchers adjust for the "dark figure of crime" by combining police records with independent victimization surveys, the apparent racial disparities in arrest rates narrow considerably. Without accounting for this reporting bias, models treat the quirks of the data collection process as objective truth, cementing systemic biases into automated software.
Similarly, in healthcare and domestic violence tracking, Karystianis et al. show that predictive models relying purely on formal reported incidents fail to generalize to the broader population of events that remain outside the view of formal legal systems.
To build robust models, researchers must stop treating public datasets as exhaustive ground truth. Recognizing the invisible filters of public reporting bias is the first step toward designing statistical corrections, integrating multi-modal data sources, and building systems that reflect the world as it is, rather than the world as it is reported.
Key Foundational Work
Prioritizing Security Spending: A Quantitative Analysis of Risk Distributions for Different Business Profiles (2015) — contributed 2 concept elements to this direction.
Data Breaches, Phishing, or Malware?: Understanding the Risks of Stolen Credentials (2017) — contributed 1 concept elements to this direction.