TL;DR
This paper introduces Metamorphic Security Testing for Web-interactions (MST-wi), a novel approach to automate security testing for web systems. By leveraging metamorphic relations (MRs) and a domain-specific language (DSL), MST-wi effectively tackles the "oracle problem" in security testing, enabling the discovery of vulnerabilities that traditional methods struggle with.
What Problem Does This Paper Address?
Modern web systems are complex, with vast input spaces and intricate interactions. Verifying their security properties is crucial but challenging due to the "oracle problem": determining the expected correct output for a given input is often infeasible, especially for security-related faults. This makes full automation of security testing difficult and costly. Existing approaches often rely on implicit oracles (like system crashes) or focus on specific vulnerability types, leaving many security testing activities unaddressed.
Methods
MST-wi employs Metamorphic Testing (MT), a technique that uses relationships between inputs and their corresponding outputs (metamorphic relations or MRs) to detect faults. Instead of defining explicit expected outputs, MST-wi checks if these relations hold.
Key innovations include:
- Security Metamorphic Relation Language (SMRL): A domain-specific language (DSL) designed for specifying security properties as MRs. It's integrated with an Eclipse editor for ease of use.
- Automated MR Transformation: SMRL specifications are automatically translated into executable Java code.
- Integrated Data Collection: MST-wi uses an extended version of the Crawljax web crawler to automatically discover system states, URLs, and actions, which are then used to derive test inputs.
- Catalog of MRs: The paper provides a catalog of 76 system-agnostic MRs covering OWASP security testing activities and CWE vulnerability types.
- Testability Guidelines: The authors propose guidelines to help engineers design systems that are more amenable to MST-wi testing.
Datasets
The paper does not explicitly mention training datasets. For evaluation, the authors used two well-known open-source web systems:
- Jenkins: A leading open-source automation server.
- Joomla: A popular open-source content management system.
The authors also reference the OWASP Top 10 and Common Weakness Enumeration (CWE) databases as sources for defining their MR catalog and analyzing vulnerability types.
Key Insights and Ideas
- Tackling the Oracle Problem: MST-wi's core strength lies in its ability to automate security testing by defining relationships between inputs and outputs, effectively bypassing the need for explicit, hard-to-define oracles.
- System-Agnostic MRs: The catalog of 76 MRs is designed to be generic, applicable to a wide range of web systems, making the approach broadly useful.
- Bridging the Gap: MST-wi effectively bridges the gap between manual security testing and full automation, particularly for vulnerabilities that are difficult to detect with traditional methods.
- Testability Matters: The research highlights that system design choices significantly impact the effectiveness of automated security testing, emphasizing the importance of testability.
Key Results
- Vulnerability Detection: MST-wi automatically detected 85% of the targeted vulnerabilities in Jenkins and Joomla.
- High Specificity: The approach demonstrated a very high specificity (99.81% on average), meaning very few false alarms were generated.
- Coverage: The MR catalog covers 39% of OWASP security testing activities not automated by other approaches and can automatically discover 102 different types of vulnerabilities (45% of those related to security design principles).
- New Vulnerability Discovery: The study led to the discovery of a new security vulnerability in Jenkins.
- Efficiency: For most vulnerability types, MST-wi can be executed overnight, and technical improvements could enable overnight execution for all MRs.
Limitations and Future Work
- Non-Web Systems: MST-wi is specifically designed for web systems and cannot directly address vulnerabilities in non-web-based systems or those relying on custom protocols.
- Program Analysis: Vulnerabilities discoverable only through program analysis (e.g., static code analysis) are outside MST-wi's scope.
- Manual Inspection: Some weaknesses require manual inspection of system outputs or data analysis, which MST-wi cannot automate.
- Third-Party Components: Weaknesses requiring control over third-party components are also challenging for MST-wi.
- Future Work: The authors suggest further research into parallelizing MR execution to improve efficiency and exploring the integration of MST-wi with other testing techniques. They also plan to expand the MR catalog and investigate its applicability to other types of systems.
Summary updated