TL;DR
This paper introduces a novel approach that combines Data Envelopment Analysis (DEA) with Automated Machine Learning (AutoML) to assess and predict a country's progress towards Sustainable Development Goals (SDGs). By integrating these methods, the study offers a more robust framework for setting national targets and improving efficiency, outperforming traditional methods like Back-Propagation Neural Networks (BPNN) in predictive accuracy.
What Problem Does This Paper Address?
The United Nations' Sustainable Development Goals (SDGs) aim to guide global progress towards a sustainable future, but tracking progress and identifying effective strategies for each country is a significant challenge. Traditional methods like Data Envelopment Analysis (DEA) are excellent for benchmarking and identifying best practices by measuring efficiency, but they lack predictive capabilities. On the other hand, machine learning (ML) can predict future trends, but designing effective ML models often requires specialized expertise and significant time. This paper addresses the need for a method that can both accurately assess current SDG performance and predict future trajectories, thereby enabling countries to set more informed and ambitious goals.
Methods
The core of this research is an integrative approach that combines DEA with AutoML. The process involves several key steps:
- Stratification DEA: This technique is used to evaluate a country's efficiency score (ES) and efficiency tier (ET), and to determine the projected output (PO) needed to transform an inefficient unit into an efficient one. Stratification helps in grouping DMUs (countries in this case) based on their performance characteristics.
- AutoML for Prediction: AutoML is then employed to predict the ES, ET, and PO. For ES and PO, regression algorithms are used, while classification algorithms are used for ET.
- BPNN Validation: A Back-Propagation Neural Network (BPNN) is used as a baseline to validate the AutoML outputs. This involves comparing the precision, accuracy, and error rates of both methods.
- DEA Re-evaluation: Finally, DEA is used again, this time with the predicted POs from AutoML and BPNN, to re-evaluate the efficiency scores and compare them, further validating the integrative approach.
The study specifically leverages TPOT (Tree-based Pipeline Optimization Tool) as the AutoML solution due to its ability to handle both regression and classification tasks and its transparent pipeline generation.
Datasets
The study utilizes two distinct datasets to demonstrate the effectiveness of the DEA-AutoML approach:
- Belt and Road Initiative (BRI) Region Data: This dataset comprises long-term, highly correlated data from 2011 to 2017 for 80 BRI countries, focusing on SDGs 7, 8, and 9 (energy, economic growth, and infrastructure).
- COVID-19 Pandemic Data: This dataset includes short-term, less correlated data related to the COVID-19 pandemic, assessing medical treatment (SDG3) and preparedness (SDG11) across 61 countries from April to May 2020.
Key Insights and Ideas
- Synergy of DEA and AutoML: The integration of DEA's benchmarking capabilities with AutoML's predictive power creates a potent tool for SDG assessment and goal-setting.
- AutoML's Superiority: AutoML consistently outperforms traditional BPNN in both regression and classification tasks for predicting SDG performance metrics.
- Robustness to Data Properties: AutoML demonstrates resilience to data characteristics that challenge BPNN, such as low standard deviation (SD) data, maintaining its predictive accuracy.
- Correlation Impact: While highly correlated data leads to higher accuracy in predictions, it doesn't significantly alter R-squared values, suggesting that model performance is influenced by more than just data correlation.
- Predictive Goals: The ability to accurately predict future SDG performance allows for the establishment of more precise and actionable national goals.
Key Results
- Regression Performance: AutoML generally achieved lower RMSE and higher PRED (percentage error less than 20%, 10%, and 5%) and R-squared values compared to BPNN across both the BRI and COVID-19 studies. For instance, in the BRI study, AutoML's R-squared values were significantly higher than BPNN's.
- Classification Performance: In classifying efficiency tiers (ET), AutoML demonstrated substantially higher accuracy (e.g., over 90% in SDG7 of the BRI study) compared to BPNN (e.g., around 30% in SDG8 of the BRI study).
- COVID-19 Preparedness: Countries generally showed good performance in preparing for COVID-19 (SDG11), with a slight downward trend in mean ES over time, indicating a widening gap between the best and worst performers. Medical treatment (SDG3) showed more variability.
- BRI Sustainability: SDG7 performance was generally good, while SDG8 progress was limited. SDG9 showed increasing inequality, with countries dividing into two distinct groups.
Limitations and Future Work
- TPOT's Multi-Output Limitation: The study notes that TPOT's current limitation in handling multi-output regression means it predicts each target variable independently, potentially overlooking interdependencies. BPNN, despite lower overall performance, handled multi-target optimization better in this specific context.
- Data Correlation: While high correlation in BRI data led to better accuracy, the R-squared values were similar to less correlated COVID-19 data, suggesting a nuanced relationship between data properties and prediction quality.
- Future Work: The authors suggest that the method's ability to reduce population size and variance in analysis can be expanded for future work, implying potential for more refined population-level insights.
Summary updated