Empirical Investigation of the Web Browser Attack Surface under Cross-Site Scripting: An Urgent Need for Systematic Security Regression Testing

Erwan Abgrall, Yves Le Traon, Sylvain Gombault, Martin Monperrus

TL;DR

This paper introduces a novel method and tool, the "XSS Test Driver," to systematically evaluate web browser vulnerability to Cross-Site Scripting (XSS) attacks. Their empirical study across a decade of browser versions reveals a concerning lack of consistent security improvement and highlights an urgent need for systematic security regression testing in browser development.

What Problem Does This Paper Address?

Cross-Site Scripting (XSS) remains a significant threat to web application security, directly targeting the end-user's web browser. While browsers have continuously evolved with new features, it's unclear if this evolution has been accompanied by systematic security testing, particularly regression testing, to ensure that new features don't introduce or reintroduce vulnerabilities. The authors aim to answer whether web browser development practices include rigorous, systematic testing against XSS attack vectors, especially considering the diverse and often obscure ways XSS attacks exploit browser mechanisms. They investigate if browser vendors maintain a consistent security policy against XSS and if their security posture improves predictably over time.

Methods

The core contribution is the development of the "XSS Test Driver," a tool designed to systematically test web browsers against a comprehensive set of XSS attack vectors. This tool executes code within the browser that mimics the behavior of an actual XSS attack, allowing researchers to measure the browser's "attack surface." The methodology involves:

  • XSS Vector Set: A curated set of 87 XSS attack vectors was compiled from various cheat sheets and augmented with newly generated vectors through systematic combination of HTML tags, properties, and JavaScript calls.
  • Browser Set: A wide range of browser families (Internet Explorer, Netscape, Mozilla, Firefox, Opera, Safari, Chrome) and their versions, spanning from 1998 to 2011, including mobile versions, were tested.
  • Testing Logic: A test case passes if the browser successfully executes the XSS payload, indicating a potential vulnerability. A test fails if the browser doesn't execute the payload, crashes, or hangs.
  • Metrics:
    • Threat Exposure Degree (ThExp): The rate of "Pass" verdicts for a given browser against a test set, indicating its overall vulnerability.
  • Degree of Noxiousness (Nox): The percentage of browsers affected by a specific XSS test case, measuring its potential impact.
  • Attack Surface Distance (ASD): The Hamming distance between the "attack surfaces" (sets of passing test results) of two browser versions, quantifying how much their behavior differs.

Datasets

The paper introduces a new benchmark for XSS testing:

  • XSS Test Driver Dataset: A set of 87 XSS attack vectors, comprising a mix of established vectors from cheat sheets and newly generated ones. This dataset is designed to be comprehensive and representative of various XSS exploitation techniques. The authors also make this dataset and the XSS Test Driver tool publicly available.

Key Insights and Ideas

  • No Universal Security Policy: Different browser families exhibit distinct attack surfaces, meaning they are not equally vulnerable to the same XSS vectors. This suggests a lack of a unified, cross-vendor security policy for XSS.
  • Chaotic Evolution of Security: The security posture of web browsers against XSS does not show a consistent trend of improvement or stabilization over time. Instead, the attack surface often fluctuates chaotically, with significant "attack surface distances" between consecutive versions.
  • Urgent Need for Regression Testing: The findings strongly indicate that browser developers are not systematically performing security regression testing for XSS. This lack of systematic testing means that vulnerabilities can reappear or persist across versions, despite ongoing development.
  • Mobile vs. Desktop Differences: While mobile and desktop versions of the same browser family often share a codebase, subtle differences in their rendering engines or event handling can lead to variations in their XSS vulnerability.
  • Legacy Browsers More Exposed: Older browsers, particularly Internet Explorer 6, show a significantly higher threat exposure to XSS compared to more modern browsers.

Key Results

  • Hypothesis H1 Validated: Browsers from different families indeed have different attack surfaces, showing varying sensitivities to XSS vectors.
  • Hypothesis H2 Validated: Web browsers are not systematically tested for XSS sensitivity, as evidenced by the lack of continuous improvement or stabilization in their attack surfaces over time.
  • Attack Surface Distance: The study revealed significant attack surface distances between browser versions, sometimes even for minor updates, highlighting instability and a lack of robust regression testing. For example, between Opera 10.50 and 10.10, the ASD was 12, despite a close number of passing vectors.
  • Browser Signatures: Each web browser exhibits a unique "signature" when subjected to the XSS benchmark, indicating that even closely related browsers can react differently to attack vectors.

Limitations and Future Work

  • Exhaustiveness of Vectors: The authors acknowledge the difficulty of creating a truly exhaustive set of XSS vectors, as new attacks can exploit obscure browser behaviors. Their attempt to generate new vectors yielded a low success rate, underscoring this challenge.
  • Test Oracle Complexity: While the XSS Test Driver aims to simplify test result interpretation, defining a definitive "oracle" for all XSS scenarios can still be complex due to the dynamic nature of web applications and browser behavior.
  • Future Work: The authors advocate for the creation of a shared security testing benchmark and propose their XSS vector set as a foundation. They suggest that their XSS Test Driver can be instrumental in ensuring systematic regression testing and opening new research avenues for test selection and browser security diagnosis. They also highlight the need for cost-efficient and automated testing methodologies.

Summary updated

Abstract

One of the major threats against web applications is Cross-Site Scripting (XSS). The final target of XSS attacks is the client running a particular web browser. During this last decade, several competing web browsers (IE, Netscape, Chrome, Firefox) have evolved to support new features. In this paper, we explore whether the evolution of web browsers is done using systematic security regression testing. Beginning with an analysis of their current exposure degree to XSS, we extend the empirical study to a decade of most popular web browser versions. We use XSS attack vectors as unit test cases and we propose a new method supported by a tool to address this XSS vector testing issue. The analysis on a decade releases of most popular web browsers including mobile ones shows an urgent need of XSS regression testing. We advocate the use of a shared security testing benchmark as a good practice and propose a first set of publicly available XSS vectors as a basis to ensure that security is not sacrificed when a new version is delivered.
BibTeX
@inproceedings{abgrall2014empirical,
  author = {Erwan Abgrall and Yves Le Traon and Sylvain Gombault and Martin Monperrus},
  title = {{Empirical Investigation of the Web Browser Attack Surface under Cross-Site Scripting: An Urgent Need for Systematic Security Regression Testing}},
  booktitle = {ICST Workshops},
  year = {2014},
  url = {https://openreview.net/forum?id=HYZb0ZanXec},
}
Figure