TL;DR
This paper introduces a novel, multi-layered index system and an improved Analytic Hierarchy Process (AHP) method to comprehensively evaluate the performance of space information network (SIN) demonstration platforms, specifically those utilizing the Tracking and Data Relay Satellite System (TDRSS). The proposed approach addresses challenges like indicator redundancy and subjective scoring, offering a more robust and consistent evaluation framework.
What Problem Does This Paper Address?
The increasing complexity and integration of space-based systems, leading to the concept of a Space Information Network (SIN), necessitate robust methods for evaluating demonstration platforms. Existing evaluation approaches face several hurdles: a vast and diverse set of technical indicators, redundancy and overlap among these indicators, difficulties in structuring indicator systems with appropriate hierarchical depth, and subjective biases in expert scoring that can lead to unreliable results. The authors aim to tackle these issues by developing a structured index system and an enhanced AHP methodology for group decision-making to ensure more consistent and accurate evaluations of SIN demonstration platforms.
Methods
The paper proposes a two-pronged approach:
- Comprehensive Index System Construction:
- Classification and Layering: A multi-step process is used, starting with investigating indicators from various SIN subsystems. Cluster analysis is employed to classify initial indicators into five criteria (coverage, timeliness, capability, accuracy, dependability).
- Hierarchical Optimization: An "element layer" is introduced between the criterion and indicator layers to create a four-tier structure, improving granularity and managing indicator outdegrees.
- Decomposition and Optimization: Indicators are further refined to ensure measurability (using coefficient of variance) and independence, removing redundant and overlapping indicators through a comprehensive weight calculation.
- Analytic Hierarchy Process (AHP) Consistency Enhancement Strategy for Group Decision-Making (AHP-GDM):
- Improved Eigenvector Method: A consistency correction strategy is proposed that uses a better initial value and exhibits faster convergence. This method leverages the Perron-Frobenius theorem and incorporates a geometric mean-induced bias matrix (GMIBM) for initial value estimation. It's designed to handle missing elements and improve the speed of convergence towards a consistent judgment matrix.
- Hadamard Convex Combination for Aggregation: To aggregate judgment matrices from multiple experts, a new weighting coefficient is introduced for the Hadamard product. This coefficient considers both expert authority and the consistency of individual judgment matrices, aiming to produce a more stable and consistent comprehensive judgment matrix.
Datasets
Not discussed in the paper. The evaluation uses simulated data derived from expert judgments for the TDRSS demonstration platform.
Key Insights and Ideas
- Structured Index System is Crucial: The paper emphasizes that a well-defined, hierarchical index system is fundamental for any comprehensive evaluation, moving beyond a simple aggregation of existing subsystem indicators.
- The "Element Layer" Improves Granularity: Introducing an intermediate layer in the index system hierarchy helps to better manage the "granularity" of indicators and the relationships between different levels, leading to a more refined evaluation structure.
- Subjectivity in AHP is a Major Hurdle: The paper highlights that expert subjectivity and discrete scaling mechanisms in traditional AHP can significantly amplify initial errors, necessitating robust consistency enhancement and aggregation strategies.
- Combining Authority and Consistency for Expert Weighting: The proposed aggregation coefficient for group decision-making is a key innovation, as it balances the influence of expert authority with the inherent consistency of their judgments, leading to more reliable group evaluations.
- Faster Convergence for Consistency: The improved eigenvector method demonstrates a significantly faster convergence rate in achieving matrix consistency compared to existing methods, which is crucial for practical applications.
Key Results
- A comprehensive evaluation index system with five criteria, 11 elements, and over 30 indicators was constructed for the TDRSS demonstration platform.
- The proposed AHP consistency enhancement strategy showed a faster convergence speed and better initial values compared to classical eigenvector methods and a confidence-based strategy.
- Numerical experiments demonstrated that the Hadamard convex combination method, incorporating expert authority and initial consistency, resulted in a better consistency ratio for the comprehensive judgment matrix.
- An evaluation example of a two-generation TDRSS system indicated that the second-generation system offers approximately 48% higher effectiveness.
Limitations and Future Work
- Dependence on Feature Vectors: The authors acknowledge the need to strengthen theoretical research on evaluation methods and try to remove dependence on feature vectors in future work.
- SIN Index System Construction: The paper suggests that future work should focus on exploring the construction of a broader SIN index system based on the presented methodology.
- Generalizability: While the method is presented as having promotion value for other network systems, further validation across diverse applications would be beneficial.
Summary updated