Optimization for massive data query method in database

Xiao Dong Xie, Jinping Zou, Xi HuangAdvances 2015 · DOI

Abstract

As database volumes expand, traditional query methods suffer from increased latency and reduced accuracy. This paper presents an optimized query method utilizing a hybrid approach of fuzzy correlation clustering and decision tree algorithms. By pre-sorting data into correlated clusters and navigating these clusters via an entropy-optimized decision tree, the proposed method maintains high accuracy even as data volume scales. Experimental results demonstrate a significant reduction in error coefficients and improved query efficiency compared to traditional algorithms.

Introduction

In the era of big data, database systems frequently encounter "data overflow," where the sheer volume and heterogeneity of information overwhelm standard retrieval mechanisms. Traditional query methods often require an excessive number of restrictions to maintain precision, leading to significant time consumption and a drop in accuracy. While standard tools like Support Vector Machines (SVM) and basic decision trees are available, they often lack the structural optimization required to handle the complex correlations inherent in massive datasets. This research proposes a dual-stage optimization strategy to address these bottlenecks.

Methodology

The proposed optimization framework consists of two primary stages: initial data organization through clustering and subsequent retrieval optimization through decision trees.

1. Fuzzy Correlation Clustering

To avoid the inefficiencies of searching through unorganized data, the system first applies fuzzy clustering. By defining a mathematical objective function, $L_p(Y, B)$, the algorithm calculates membership degrees and cluster centers. This iterative process identifies hidden correlations between data points, effectively "pre-sorting" the database into meaningful categories. This fuzzy approach allows data points to maintain relationships with multiple categories, providing the flexibility needed for complex queries.

2. Decision Tree Optimization

Once the data is clustered, the system constructs an optimized decision tree to navigate queries. The construction is guided by the Information Gain Ratio ($I_t(C_h)$). By selecting attributes with the highest information gain as branch nodes, the system ensures that the search space is narrowed down as rapidly as possible, minimizing the number of logical checks required to locate specific data.

Hierarchical structure of the optimized decision tree for database navigation.

Experimental Results and Analysis

The performance of the proposed method was evaluated against a traditional query algorithm using a simulated massive database. The primary metric for evaluation was the Error Coefficient, measured as the data size scaled from 100 to 500 units.

As shown in the experimental data, the traditional algorithm's error coefficient increases dramatically as the database grows, rising from approximately 0.1 to 0.8. In contrast, the proposed method exhibits remarkable stability. Its error coefficient remains consistently low, starting near 0.15 and only reaching approximately 0.25 at the highest data volume tested. This suggests that the hybrid clustering-decision tree approach is significantly more resilient to the challenges of data overflow.

Comparison of error coefficients showing the stability of the proposed algorithm versus traditional methods as data size increases.

Conclusion

The integration of fuzzy correlation clustering with optimized decision trees provides a robust solution for querying massive databases. By leveraging the mathematical strengths of entropy-based branching and the flexibility of fuzzy logic, the proposed method achieves higher accuracy and better scalability than traditional approaches. Future research may focus on the computational overhead of the initial clustering phase and the adaptation of this model for real-time streaming data environments.

Summary updated

Abstract

The massive data query methods in database is studied to improve accuracy of query. In the process of data querying in the database, once the data volume is overflow and thetype of data becomes complex, the query requires a lot of restrictions, resulting in time-consuming and low query accuracy rate for data query. To this end, optimization for massive data query method in database based oncorrelation clustering algorithm is proposed. Correlation clustering is processed to all the data in the database to obtain the correlation between different data. Query for the specified categories of data to achieve query optimization of massive data in database. Experimental results show that the proposed method for mass data query in database can improve the accuracy and efficiency of the query, and reduce time for querying.
BibTeX
@inproceedings{xie2015optimization,
  author = {Xiao Dong Xie and Jinping Zou and Xi Huang},
  title = {{Optimization for massive data query method in database}},
  booktitle = {Advances},
  year = {2015},
  doi = {10.2991/amcce-15.2015.272},
}
Figure