Better Safe Than Sorry! Automated Identification of Functionality-Breaking Security-Configuration Rules

Patrick Stöckle, Michael Sammereier, Bernd Grobauer, Alexander Pretschner

TL;DR

This paper introduces an automated approach to identify security configuration rules that break existing system functionality, a major hurdle for administrators adopting security best practices. By combining combinatorial testing with machine learning (decision trees) and logic minimization, they can pinpoint problematic rules, allowing for safer system hardening without sacrificing functionality.

What Problem Does This Paper Address?

Many systems ship with insecure default configurations, leaving them vulnerable to attacks. Security configuration guides, like those from the Center for Internet Security (CIS), offer detailed recommendations for hardening systems. However, system administrators are often hesitant to apply these rules due to a fear of breaking existing functionalities. This fear is well-founded, as blindly applying hundreds of rules can indeed lead to unexpected system failures. The problem is that identifying which specific rules cause these issues is a complex, time-consuming, and often manual process, making it impractical for many organizations. This paper aims to automate this identification process, enabling administrators to apply security configurations confidently.

Methods

The core of the proposed solution lies in a two-pronged approach:

  1. Combinatorial Testing (Covering Arrays): Instead of testing every possible combination of security rules (which is computationally infeasible), the authors leverage combinatorial testing. They generate "covering arrays" which represent a minimal set of test configurations that cover all pairwise (or higher-order) interactions between rules. This significantly reduces the number of test cases needed. Algorithms like IPOG and IPOG-D are used for generating these arrays.
  2. Machine Learning and Logic Minimization: After applying these rule combinations to a system and observing which ones break functionality (using automated tests), the results are fed into analytical models.
  3. Decision Trees: A machine learning approach where a decision tree is trained on the test results. Each node in the tree represents a security rule, and the paths from the root to the leaves indicate combinations of applied/excluded rules and whether they resulted in a broken functionality. Shortest path algorithms (like Dijkstra's) are then used on a weighted decision tree to find the maximal set of non-breaking rules.
  4. Logic Minimization: A more formal approach that uses the test results to build a truth table. Logic minimization algorithms are then applied to derive a minimal set of rules that, when excluded, ensure the system's functionality.

Datasets

The paper does not introduce new datasets for training machine learning models. Instead, it uses a CIS guide for Windows 10 (version 1909) with 507 rules as the basis for their evaluation. For testing, they either simulate breaking rules or use a simple PowerShell script that fails when the entire guide is applied.

Key Insights and Ideas

  • Bridging the Gap: The paper effectively bridges the gap between security best practices (configuration guides) and practical system administration by addressing the critical issue of functionality breakage.
  • Efficiency through Combinatorics: Using combinatorial testing drastically reduces the testing effort compared to brute-force methods, making the process more scalable.
  • Automated Rule Identification: The combination of combinatorial testing and analytical models (decision trees/logic minimization) provides an automated way to identify problematic rules, saving significant manual effort.
  • The "Fear Factor" is Real: The paper validates the administrators' concerns, highlighting that even simple rules can break complex functionalities, and that a significant percentage of default configurations are non-compliant with security guides.
  • Need for Automated Testing: A recurring theme is the critical dependency on good, automated tests to make this approach viable. The lack of such tests in many organizations is a significant bottleneck.

Key Results

  • The approach successfully identified breaking rules in their test scenarios. For instance, in a practical application, rule R1_1_4 ("ensure 'Minimum password length' is set to '14 or more characters'") was identified as breaking a test script that tried to set a password of length 6.
  • The time to generate covering arrays varied significantly with strength and algorithm, with IPOG-D being faster than IPOG. Generating strength 5 covering arrays for a 507-rule guide took approximately 13.7 days.
  • The testing process, including VM setup, rule application, and testing, took around 12 hours for strength 4 covering arrays.
  • The analysis phase (decision tree or logic minimization) was relatively fast, taking seconds to minutes for heuristic approaches.
  • The heuristic approach (decision trees) achieved a success rate of 77% on a complex sample set, and close to 100% on more realistic scenarios (assuming breaking rules involve 4 or fewer rules). Logic minimization performed similarly but took longer.

Limitations and Future Work

  • Strength of Covering Arrays: The authors assumed a strength of 6 for covering arrays but found it computationally infeasible to generate for large guides. They suggest that 4-way covering arrays are more realistic, which might miss breaking rules involving 5 or more rules.
  • Lack of Real-World Breaking Rule Sets: The breaking rule sets used in their evaluation were generated based on literature claims and might not perfectly represent real-world problems, which could be system-dependent and more complex. A case study with real-world configuration problems is suggested.
  • Dependency on Automated Tests: The entire approach hinges on the availability of good, automated tests that can reliably detect functionality breakage. The paper acknowledges that this is a significant limitation in many organizations.
  • Importance of Rules: The current approach treats all rules equally. Future work could incorporate the importance or criticality of rules to prioritize hardening efforts.
  • Maximal vs. Practical Sets: While the goal is a maximal non-breaking set, a practical subset might be sufficient and more economically viable.
  • System Isolation: The assumption of being able to isolate a system in an image might not hold for all environments, especially complex, interconnected systems.

Summary updated

Abstract

Insecure default values in software settings can be exploited by attackers to compromise the system that runs the software. As a countermeasure, there exist security-configuration guides specifying in detail which values are secure. However, most administrators still refrain from hardening existing systems because the system functionality is feared to deteriorate if secure settings are applied. To foster the application of security-configuration guides, it is necessary to identify those rules that would restrict the functionality. This article presents our approach to use combinatorial testing to find problematic combinations of rules and machine learning techniques to identify the problematic rules within these combinations. The administrators can then apply only the unproblematic rules and, therefore, increase the system's security without the risk of disrupting its functionality. To demonstrate the usefulness of our approach, we applied it to real-world problems drawn from discussions with administrators at Siemens and found the problematic rules in these cases. We hope that this approach and its open-source implementation motivate more administrators to harden their systems and, thus, increase their systems' general security.
BibTeX
@inproceedings{stockle2023better,
  author = {Patrick Stöckle and Michael Sammereier and Bernd Grobauer and Alexander Pretschner},
  title = {{Better Safe Than Sorry! Automated Identification of Functionality-Breaking Security-Configuration Rules}},
  booktitle = {Ast},
  year = {2023},
  doi = {10.1109/ast58925.2023.00013},
}
Figure