TL;DR
Designing secure software systems is challenging due to the inherent uncertainty and complexity of cyber threats. This paper introduces a systematic approach to measure and aggregate expert opinions on security risks, aiming to provide more consistent and reliable advice for system designers. By quantifying expert judgments and accounting for their variability, the research offers a method to build a more robust understanding of system vulnerabilities.
What Problem Does This Paper Address?
The cybersecurity landscape is constantly evolving, with new attack vectors and technologies emerging rapidly. This makes it incredibly difficult for security experts to provide consistent and reliable assessments of system security, especially during the design phase. Traditional methods often rely on subjective expert opinions, which can vary significantly between individuals due to differences in experience, knowledge, and even personality. This variability can lead to inconsistent security advice, making it hard for system designers to make informed decisions. The paper aims to bridge this gap by developing a systematic methodology to elicit, measure, and aggregate these expert opinions, thereby reducing the impact of individual inconsistencies and producing a more coherent consensus view of security risks.
Methods
The core of the paper's methodology involves eliciting and analyzing expert opinions through structured exercises. Initially, experts were asked to rank "attack vectors" (AVs) by difficulty and then rate individual "hops" (components of an attack) on a simple scale. To capture more nuanced subjective uncertainty, a novel approach was developed using interval responses. Experts provided their answers as ellipses on a scale, where the width of the ellipse represented their confidence or uncertainty. This allowed for a more detailed differentiation of difficulty and certainty. For data analysis, various aggregation techniques were employed, including simple statistical measures (mean, maximum, minimum) and more sophisticated methods like Ordered Weighted Average (OWA) operators. These operators were used to combine individual hop ratings into an overall difficulty score for attack vectors, aiming to reduce ties and produce more meaningful rankings.
Datasets
The paper describes data collected from two main exercises:
- Initial Exercise: Involved six experienced cybersecurity experts from CESG.
- Main Exercise: Involved thirty-nine security professionals from seven different groups (government and commercial backgrounds), including system/software architects, security consultants, penetration testers, vulnerability researchers, and specialist systems evaluators.
The scenario used in both exercises was a representative mainstream government system, designed by a senior CESG technical staff member. The data collected included:
- Rankings of ten attack vectors (AVs) by difficulty.
- Ratings of twenty-six distinct "hops" (attack components) across various dimensions, including overall difficulty and confidence, using interval responses.
Key Insights and Ideas
- Expert Opinion is Crucial but Variable: The research confirms that expert opinion is essential for assessing security risks, especially in complex and uncertain environments. However, it also highlights the significant variability in these opinions, which needs to be managed.
- Aggregation Reduces Uncertainty: Aggregating expert opinions, even from a relatively small group, can lead to a more consistent and reliable assessment of security risks than relying on individual judgments.
- Interval Responses Capture Nuance: The novel use of interval responses (ellipses) proved effective in capturing not just the perceived difficulty but also the expert's confidence or uncertainty, providing richer data for analysis.
- OWA Operators Improve Ranking: Ordered Weighted Average (OWA) operators, which give more weight to more difficult hops, were found to be superior to simple maximum or minimum operators for aggregating hop ratings, as they reduced ties and produced more meaningful attack vector rankings.
- Identifying Outliers is Valuable: The methodology allows for the identification of "outlier" experts whose opinions significantly deviate from the norm. This can be valuable for identifying individuals with cutting-edge knowledge or those who may require additional training.
Key Results
- The initial exercise with six experts showed a high degree of correlation between individual rankings and the group rank, and strong correlation among the experts themselves (Kendall's W of 0.82), suggesting a consensus on attack vector difficulty.
- The main exercise with thirty-nine experts demonstrated that while individual opinions varied, aggregating group opinions resulted in a very strong consensus, producing a consistent measure of attack vector difficulty.
- Analysis of hop data showed a strong correlation between an individual's hop ratings and their overall attack vector rankings, indicating that the perceived difficulty of constituent hops strongly influences the perceived difficulty of the entire attack vector.
- The study successfully identified individuals with particularly strong or weak correlations to the overall group consensus, highlighting the potential for targeted training or knowledge dissemination.
Limitations and Future Work
- "Ground Truth" is Subjective: The paper acknowledges that consensus opinion is not necessarily "ground truth" but rather a strong indicator of likely truth if expertise and agreement are high.
- Outlier Interpretation: While outliers can be identified, their significance (cutting-edge knowledge vs. lack of experience) requires careful management and further investigation.
- Optimal Number of AVs: The optimal number of attack vectors to use for a given system's complexity requires further experimentation.
- Future Hop Data Analysis: Future work will involve a more comprehensive analysis of the hop data, including all experts, exploring remaining questions about hops, analyzing expressed uncertainties, and determining consistencies between experts.
- Real-World Applications: The authors aim to explore real-world applications, such as developing a validation tool for system designers and establishing a consensus opinion mechanism.
Summary updated