The Research of Meteorological Data Mining Using Discrete Bayesian Networks Classifier Based on Hadoop

Yongjun Zhang, Jing SunAdvances 2015 · DOI

Abstract

The method of Native Bayesian classification data mining in weather forecast has some defects, such as there is not independent of each other between predictors, but a certain relevance which results in the decrease of prediction accuracy. This paper explores an improved algorithm which is based on the theory of discrete Bayesian Networks, and combines with Hadoop distributed file system and parallel processing programming models to predict rainfall. The experiments show that the improved algorithm not only makes the classification prediction more reliable but also improves the efficiency greatly. In addition, it provides a solution of huge amounts of data mining in the other fields.

Introduction: The "Independence Trap"

Weather forecasting is a classic "Big Data" challenge, but traditional tools often struggle with two main issues: accuracy and scale. Most simple Bayesian classifiers are "Naive"—they assume that every predictor (like air pressure, temperature, or humidity) is completely independent of the others. In the real world, these variables are deeply interconnected; temperature affects pressure, which in turn affects humidity. Ignoring these correlations leads to poor predictions. Furthermore, as meteorological datasets grow into the terabyte range, single-machine processing becomes a bottleneck. This research aims to bridge the gap between complex statistical modeling and high-performance distributed computing.

Methodology: Discrete Bayesian Networks and Hadoop

The authors propose a two-part solution to overcome these limitations:

  1. Discrete Bayesian Networks (DBN): Unlike Naive Bayes, DBNs model the dependencies between variables using a directed acyclic graph. The authors use Predictive Ability Theory to determine which "arcs" (connections) should exist between variables. If one variable significantly helps predict another, a connection is drawn, allowing the model to capture the nuanced relationships of the atmosphere.
The initial (a) and improved (b) Bayesian Network structures showing variable dependencies.
  1. Hadoop/MapReduce Integration: To handle massive datasets, the entire pipeline is parallelized. MapReduce jobs find the min/max values of predictors and discretize them into intervals. Parallel tasks then calculate the Statistical Frequency and Conditional Probability Tables (CPT) for the network across the cluster.

Experimental Setup and Results

The study utilizes historical data from the China Meteorological Data Sharing Service System, spanning from 1951 to 2014. Features include average pressure, precipitation, average temperature, average vapor pressure, wind speed, and sunshine time.

Prediction Accuracy

The model was tested across four categories: No Rain (R0), Light Rain (R1), Moderate Rain (R2), and Heavy Rain (R3). The Discrete Bayesian Network significantly outperformed standard Naive Bayesian methods, particularly in identifying "No Rain" scenarios.

Comparison of correct rate (a) and predicted rate (b) between Native Bayesian and the proposed Bayesian Network.

Key findings include:

  • An impressive 88.68% correct rate for "No Rain" (R0) scenarios.
  • While accuracy dropped for heavier rainfall (38.1% for R3), it remained consistently higher than the Naive Bayesian baseline.

Efficiency and Scalability

The most important takeaway for engineers is the system's linear scalability. In tests using a 50GB dataset, increasing the Hadoop cluster from 3 nodes to 10 nodes slashed the processing time from approximately 5,500 seconds to under 2,000 seconds. This proves that Bayesian Networks—often computationally expensive—can be effectively "tamed" using distributed frameworks.

Conclusion

By explicitly modeling the correlation between predictors and leveraging the parallel processing power of Hadoop, this approach makes rainfall prediction both more reliable and more efficient. While the discretization of continuous weather data can lead to some loss of granular information, the system provides a robust solution for mining massive, correlated datasets in meteorology and beyond.

Summary updated

Abstract

The method of Native Bayesian classification data mining in weather forecast has some defects, such as there is not independent of each other between predictors, but a certain relevance which results in the decrease of prediction accuracy. This paper explores an improved algorithm which is based on the theory of discrete Bayesian Networks, and combines with Hadoop distributed file system and parallel processing programming models to predict rainfall. The experiments show that the improved algorithm not only makes the classification prediction more reliable but also improves the efficiency greatly. In addition, it provides a solution of huge amounts of data mining in the other fields.
BibTeX
@inproceedings{zhang2015research,
  author = {Yongjun Zhang and Jing Sun},
  title = {{The Research of Meteorological Data Mining Using Discrete Bayesian Networks Classifier Based on Hadoop}},
  booktitle = {Advances},
  year = {2015},
  doi = {10.2991/icecee-15.2015.189},
}
Figure