Abstract: If a person looks at WHITE paper through BLUE glasses, the paper will become BLUE in the eye of the person. Likewise, in the current study of big data which play the same role as the white paper being looked at, various statistical methods just serve as the blue glasses. That is, results obtained from big data often depend on the statistical methods in use, which may often defy reality. Here I suggest using physical ideas and methods to overcome this problem to the greatest extent. This suggestion is helpful to development and application of big data.
Introduction: The "Blue Glasses" of Big Data
The current landscape of big data analysis is often dominated by purely statistical methods. While these tools are powerful for identifying patterns, they can act as "blue glasses" that distort the underlying reality of the "white paper" (the data). Results obtained through these filters often depend more on the specific statistical method in use than on the actual phenomena being studied, leading to conclusions that may defy reality. To move beyond misleading correlations, researchers must adopt the physical paradigm: a three-tiered approach combining empirical analysis, controlled experiments, and theoretical modeling to uncover true causal mechanisms.
The Core Framework: Physical Ideas (PIs)
The physical paradigm relies on two fundamental ideas (PIs) that have historically guided scientific discovery. The first is coarse-graining, the process of extracting only the crucial factors while neglecting the "noise" of extraneous variables. This is exemplified by Galileo's decision to ignore air resistance to uncover the law of free fall. The second is universality, ensuring that results not only explain historical data but also possess the power to predict unknown phenomena, much like Newton's laws predicted the existence of Neptune.
The Methodological Hierarchy: Physical Methods (PMs)
To implement these ideas, we categorize scientific research into three levels of Physical Methods (PMs).
- PM 1 (Empirical Analysis): This level involves observing and analyzing natural phenomena to reveal correlations. Most current big data research resides here.
- PM 2 (Controlled Experiments): This level moves beyond observation by purposefully tuning specific parameters while holding others constant. This reveals cause-effect relationships (causations).
- PM 3 (Theoretical Analysis): This is the highest level, where mathematical tools (like calculus) are used to generalize experimental results into universal laws.
Ideally, "Big Data Science" should be the application of PM 3 to massive datasets to bridge the gap between structure (volume) and function (predictive value).
From Correlation to Causation: Case Studies
The limitations of pure empirical analysis (PM 1) are well-documented. For instance, Aristotle’s geocentric model was a logical conclusion of PM 1 that was eventually overturned by the PM 2 and PM 3 approaches of Galileo and Newton.
In modern finance, traditional PM 1 analysis suggests a positive risk-return relationship. However, when researchers applied PM 3—combining controlled experiments with theoretical modeling—they discovered Bowman’s paradox: a negative relationship in efficient, closed markets. Similarly, Adam Smith’s "invisible hand" has been validated by modern agent-based simulations (PM 3), providing a mechanistic explanation for market equilibrium that pure observation alone could not fully substantiate.
Challenges and Future Directions
Adopting this physical paradigm is not without hurdles. In fields like astrophysics or certain social sciences, controlled experiments (PM 2) are often impossible, necessitating a reliance on PM 1 as a compromise. Furthermore, there is no universal manual for PM 3; theoretical analysis must be tailored to the specific context of the data.
Future work should focus on moving beyond simple statistical screening toward building robust mathematical models that can be tested against controlled experimental data within the big data ecosystem. The goal is to ensure that data is not just massive, but "valuable" in its ability to explain the past and predict the future.
Conclusion
Big data science must evolve from a collection of correlations into a rigorous discipline that seeks mechanisms. By adopting the physical paradigm—integrating coarse-graining, universality, and the hierarchy of physical methods—researchers can move beyond the "blue glasses" of statistics to uncover the true causal structures underlying complex datasets.
Summary updated