How did Donald Trump Surprisingly Win the 2016 United States Presidential Election? an Information-Theoretic Perspective (Clean Sensing for Big Data Analytics: Optimal Strategies, Estimation Error Bounds Tighter than the Cramér-Rao Bound)

Weiyu Xu, Lifeng Lai, M. Amin KhajehnejadarXiv 2018 · OpenReview

Introduction

The 2016 US Presidential Election remains a watershed moment for data science and political polling. Despite nearly all major opinion polls predicting a victory for Hillary Clinton, Donald Trump secured the presidency. This discrepancy raised fundamental questions about the accuracy of modern polling and the limits of big data analytics. This paper introduces "clean sensing," an information-theoretic framework designed to address systematic distortions in heterogeneous data sources, providing a mathematical explanation for why more data does not always lead to better results.

The 2016 Polling Failure

The failure of 2016 forecasts highlighted a critical flaw: when data contains systematic distortions—such as the "shy Trump voter" effect where respondents may not truthfully disclose their preferences—simply increasing the sample size fails to reduce estimation error.

Comparison of 2016 Polling Averages vs. Final Results across key states.

As shown in the table above, states like Michigan, Pennsylvania, and Wisconsin saw final results that deviated significantly from polling averages, often flipping the predicted winner. The authors argue that these errors are not merely statistical noise but systematic biases that traditional polling methods fail to mitigate.

The Clean Sensing Framework

To address this, the authors propose "clean sensing" (or clean polling), a general framework for optimal parameter estimation under budget constraints. They model the polling process as a discrete memoryless channel where a voter's true preference passes through a "distortion channel."

Crucially, the model assumes that the probability of distortion (e.g., a voter lying) can be reduced by increasing the "sensing cost" per sample—for instance, through more rigorous verification or more time-intensive interviewing techniques. The goal is to minimize the Cramér-Rao lower bound (CRB) on the mean-squared error (MSE) of the estimator, subject to a total cost budget.

Optimal Polling Strategies: Quality Over Quantity

Using Lagrange dual problems and Karush-Kuhn-Tucker (KKT) conditions, the paper derives several key insights that challenge conventional wisdom:

  • Big Data Limitations: The authors mathematically prove that if an estimator uses simple averaging on distorted data, the error will never reach zero, regardless of sample size.
  • Resource Allocation: The optimal strategy involves allocating budget based on a "cost-over-Fisher-information" ratio. If a specific demographic is highly distorted, the optimal approach is actually to poll fewer individuals from that group but spend significantly more resources on each to ensure high data quality.
  • Sample Size Optimization: The optimal number of samples from a group should be inversely proportional to the square root of the product of Fisher information and cost.

Beyond Polling: New Statistical Lower Bounds

The paper's contributions extend into pure statistical theory. The authors frame the search for MSE lower bounds as a convex optimization problem. From this, they derive three new series of lower bounds (Series A, B, and C) for unbiased and biased parameter estimators.

These new bounds are significant because they can be strictly tighter than the classical Cramér-Rao and Chapman-Robbins bounds. In a toy example involving a Bernoulli distribution, the authors demonstrate that their "Series A" bound correctly identifies an infinite MSE where the Chapman-Robbins bound incorrectly suggests a finite limit. This suggests that the classical bounds may sometimes offer an overly optimistic view of estimator performance.

Conclusion

The "clean sensing" framework provides a rigorous mathematical foundation for understanding the limitations of big data in the presence of systematic bias. By shifting the focus from the quantity of data to the optimal balance of quality and quantity, this research offers a path toward more resilient polling and data analytics. While implementing these strategies requires knowledge of cost-distortion functions, the theoretical insights provide a crucial corrective to the "more is always better" philosophy of the big data era.

Summary updated

Abstract

Donald Trump was lagging behind in nearly all opinion polls leading up to the 2016 US presidential election, but he surprisingly won the election. This raises the following important questions: 1) why most opinion polls were not accurate in 2016? and 2) how to improve the accuracies of opinion polls? In this paper, we study the inaccuracies of opinion polls in the 2016 election through the lens of information theory. We first propose a general framework of parameter estimation, called clean sensing (polling), which performs optimal parameter estimation with sensing cost constraints, from heterogeneous and potentially distorted data sources. We then cast the opinion polling as a problem of parameter estimation from potentially distorted heterogeneous data sources, and derive the optimal polling strategy using heterogenous and possibly distorted data under cost constraints. Our results show that a larger number of data samples do not necessarily lead to better polling accuracy, which give a possible explanation of the inaccuracies of opinion polls in 2016. The optimal sensing strategy should instead optimally allocate sensing resources over heterogenous data sources according to several factors including data quality, and, moreover, for a particular data source, it should strike an optimal balance between the quality of data samples, and the quantity of data samples. As a byproduct of this research, in a general setting, we derive a group of new lower bounds on the mean-squared errors of general unbiased and biased parameter estimators. These new lower bounds can be tighter than the classical Cram\'{e}r-Rao bound (CRB) and Chapman-Robbins bound. Our derivations are via studying the Lagrange dual problems of certain convex programs. The classical Cram\'{e}r-Rao bound and Chapman-Robbins bound follow naturally from our results for special cases of these convex programs.
BibTeX
@misc{xu2018how,
  author = {Weiyu Xu and Lifeng Lai and M. Amin Khajehnejad},
  title = {{How did Donald Trump Surprisingly Win the 2016 United States Presidential Election? an Information-Theoretic Perspective (Clean Sensing for Big Data Analytics: Optimal Strategies, Estimation Error Bounds Tighter than the Cramér-Rao Bound)}},
  year = {2018},
  eprint = {1812.11891},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1812.11891},
}

Loading related content...

Figure