Caveat emptor: the risks of using big data for human development

Siddique Latif, Adnan Qayyum, Muhammad Usama, Junaid Qadir, Andrej Zwitter, Muhammad ShahzadarXiv 2019 · arXiv

TL;DR

While Big Data for Development (BD4D) promises to revolutionize how we tackle poverty, health, and crises, it is a double-edged sword. This paper provides a critical reality check, arguing that "data worship" blinds us to significant risks—ranging from systemic biases and the erosion of privacy to the "Cobra effect" of unintended consequences—and proposes a framework for more responsible, interpretable, and human-centric analytics.

What Problem Does This Paper Address?

The "data deluge" has led many to view big data as a silver bullet for sustainable human development. However, the authors argue that there is a dangerous gap between the hype and the reality of deploying these systems in sensitive social contexts.

Decision-makers are increasingly swapping intuition for data-driven evidence, but this evidence is often built on "soft" numbers, biased samples, and opaque "black-box" models. When applied to human development, these technical flaws don't just result in bad code; they result in "weapons of maths destruction" that can erode privacy, punish individuals for their propensities rather than actions, and exacerbate social sectarianism. The paper seeks to move the conversation from uncritical enthusiasm to a "caveat emptor" (buyer beware) approach.

Methods

This paper is a critical synthesis and position paper. Rather than introducing a new algorithm, the authors perform an interdisciplinary review of existing literature across computer science, statistics, sociology, and economics.

They categorize the risks of big data into four primary pillars:

  1. Epistemological Issues: The reductionist nature of data and the myth of objectivity.
  2. Technical Pitfalls: Opaque models, sampling biases, and the "curse of dimensionality."
  3. Development-Specific Challenges: The "statistical tragedy" of unreliable national data and the complexity of human feedback loops.
  4. Ethical/Privacy Risks: The failure of traditional anonymization and the potential for malicious use.

Finally, they map out a set of "remedies," drawing on techniques like Interpretable AI (LIME, LRP), causal analysis, and statistical corrections.

Datasets

As a review and position paper, it does not introduce a new training dataset. Instead, it analyzes several high-profile real-world case studies to illustrate its points:

  • Google Flu Trends: Used to demonstrate "activity bias," where the model over-predicted flu cases due to changes in user search behavior rather than actual disease spread.
  • African National Statistics: Specifically the GDP recalculations in Ghana (2010) and Nigeria (2014), which saw overnight "growth" of 60% and 90% respectively, highlighting the unreliability of the underlying data used for policy.
  • Mobility Datasets: Referencing studies showing that just four spatiotemporal points are enough to uniquely identify 95% of individuals in a "de-identified" dataset.

Key Insights and Ideas

  • The Myth of Neutrality: Data is not "baggage-free." Every model is an "opinion embedded in math." The authors invoke the "No Free Lunch" theorem to remind us that there are no universally applicable solutions without subjective trade-offs.
  • Reductionism vs. Reality: Big data often ignores the concrete contextual realities of human societies, reducing complex lives to simple, granular outputs that are difficult to translate back into real-world implementations.
  • The Cobra Effect: In complex human systems, interventions often have unintended consequences. The authors cite the historical example of a bounty on cobras in colonial India that led people to farm cobras for profit, ultimately increasing the snake population.
  • Africa’s Statistical Tragedy: In many low-income countries, "official" statistics are often inferred from proxies like rainfall or population growth rather than direct measurement, leading to a "farce" of self-monitoring by states.
  • Anonymization is Fragile: The paper highlights that 87% of Americans can be uniquely identified using only three bits of information: zip code, birth date, and sex.

Key Results

While the paper is qualitative, it highlights several striking quantitative findings from the literature it reviews:

  • Economic Impact of Bias: Organizations that actively work to minimize bias in decision-making achieve up to 7% higher returns.
  • Privacy Vulnerability: Spatiotemporal data is so unique that four points are sufficient to identify the vast majority of a population.
  • Data Growth vs. Signal: While data increases by 2.5 quintillion bytes daily, the "noise" is increasing faster than the "signal," leading to a plethora of spurious correlations.

Limitations and Future Work

The authors acknowledge that while they propose remedies, many are still in their infancy:

  • Interpretable AI for BD4D: Applying tools like LIME or SHAP to high-stakes development decisions remains an "open research avenue."
  • Humanitarian Intelligence: There is a need to develop better methods for corroborating big data findings with "boots on the ground" field intelligence.
  • Regulatory Gaps: The paper calls for new laws and industry standards, such as the "right to be forgotten," to give individuals control over how their data influences their lives.
  • Causality: The field needs to move beyond correlation toward a mathematical framework for causality (referencing Judea Pearl’s work) to be truly useful for policy.

Summary updated

Abstract

Big data revolution promises to be instrumental in facilitating sustainable development in many sectors of life such as education, health, agriculture, and in combating humanitarian crises and violent conflicts. However, lurking beneath the immense promises of big data are some significant risks such as (1) the potential use of big data for unethical ends; (2) its ability to mislead through reliance on unrepresentative and biased data; and (3) the various privacy and security challenges associated with data (including the danger of an adversary tampering with the data to harm people). These risks can have severe consequences and a better understanding of these risks is the first step towards mitigation of these risks. In this paper, we highlight the potential dangers associated with using big data, particularly for human development.
BibTeX
@misc{latif2019caveat,
  author = {Siddique Latif and Adnan Qayyum and Muhammad Usama and Junaid Qadir and Andrej Zwitter and Muhammad Shahzad},
  title = {{Caveat emptor: the risks of using big data for human development}},
  year = {2019},
  eprint = {1905.00493},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1905.00493},
}
Figure