A Comprehensive and Cross-Platform Test Suite for Memory Safety - Towards an Open Framework for Testing Processor Hardware Supported Security Extensions

Wei Song, Jiameng Ying, Sihao Shen, Boya Li, Hao Ma, Peng LiuarXiv 2021 · arXiv

Abstract

Memory safety vulnerabilities remain a persistent threat in systems programmed with low-level languages like C and C++. While hardware-supported security extensions, such as Intel Control-flow Enforcement Technology (CET) and Arm Memory Tagging Extension (MTE), offer promising low-overhead defenses, their effectiveness across diverse platforms remains difficult to quantify. This paper introduces an open-source, expandable framework and a comprehensive test suite of 160 cases designed to evaluate memory safety defenses across various processor architectures, operating systems, and compilers. By utilizing a novel relation graph to map vulnerability dependencies, the framework provides a standardized methodology for measuring security coverage and identifying gaps in modern hardware-software stacks.

Introduction

Memory safety remains the "eternal war" of computer security. Low-level languages like C and C++ are prone to bugs—such as buffer overflows or use-after-free errors—that allow attackers to hijack systems. While many software-based defenses exist, they often impose a heavy performance penalty, leading developers to disable them in production.

Recently, chipmakers have introduced hardware-supported defenses to provide security with minimal overhead. However, there hasn't been a consistent way to test if these features actually work as advertised across different platforms. Existing test suites are often outdated, limited to specific architectures (like x86), or fail to distinguish which specific part of the system (the hardware, the kernel, or the compiler) is actually providing the protection. This work addresses these gaps by providing a portable, comprehensive, and logically structured testing framework.

Methodology

The authors' framework is built on three main pillars: a logical relation graph, a cross-platform architecture, and anti-optimization techniques.

The Relation Graph

Instead of treating vulnerabilities as a flat list, the authors created a graph that defines the relationships between them. This includes dependencies (e.g., a Return-Oriented Programming (ROP) attack might depend on bypassing ASLR), specialization, and relaxation. This allows the suite to determine "coverage" logically—if a "prerequisite" defense fails, the suite knows which subsequent attacks are likely to succeed.

The relation graph for ROP attacks illustrates the logical dependencies and relaxations between different vulnerability types.

Cross-Platform Architecture and Portability

To ensure the suite works on x86, Arm, and RISC-V, the authors separated the tests into platform-independent C++ code and a small, shared library of platform-specific assembly macros. This modularity allows the framework to be easily extended to new architectures as they emerge.

The framework utilizes a wrapper to include architecture-specific assembly headers, ensuring high portability across x86, Arm, and RISC-V.

Anti-Optimization Techniques

Compilers are often "too smart" for security testing; they frequently optimize away the intentional bugs used in test cases. The authors used specific compiler attributes, such as volatile variables and noinline functions, alongside embedded assembly to ensure the vulnerabilities remain exploitable for testing purposes.

The Memory Safety Test Suite

The primary contribution is a suite of 160 specialized test cases, categorized into five key areas:

  • Spatial Safety (98 cases): Covers buffer overflows and underflows across the stack, heap, and global data.
  • Temporal Safety (13 cases): Focuses on use-after-free and double-free vulnerabilities.
  • Access Control (3 cases): Evaluates the effectiveness of ASLR and page permissions.
  • Pointer Integrity (5 cases): Tests protections for function pointers and VTables.
  • Control-Flow Integrity (41 cases): Targets ROP, Jump-Oriented Programming (JOP), and VTable injection.

The suite was evaluated on six distinct platforms, including Intel i7/Xeon, Nvidia Jetson Nano (Arm), Raspberry Pi 4B (Arm), and HiFive Unleashed (RISC-V).

Key Insights and Results

The evaluation revealed several critical insights into the current state of hardware-supported security:

  • The "Default" Vulnerability: On almost all tested platforms, the "default" configuration provides very little protection. Many hardware features are present but remain unused because the OS or compiler hasn't enabled them. Across all six platforms, zero defenses were detected by default against spatial safety attacks.
  • RISC-V Maturity: The RISC-V platform (using the Spike simulator) showed marginal security benefits due to its support for "Read-Exclusive Execution" (R⊕X) in page tables, which thwarts certain code-injection attacks.
  • ASan's Dominance: While hardware features are the future, the software-based AddressSanitizer (ASan) remains the most effective defense in the suite, catching nearly all spatial and temporal bugs, though it is usually too slow for production use.
  • Compiler Flags: Enabling the RELRO (Relocation Read-Only) flag successfully protected the Global Offset Table (GOT) on most platforms, but other flags like Stack Protection provided very limited defense against sophisticated ROP attacks.

Limitations and Future Work

While comprehensive, the framework has room for growth. Current limitations include a lack of support for multi-threaded concurrency vulnerabilities and Data-Oriented Programming (DOP) attacks. Furthermore, the framework does not yet test for side-channel or transient execution attacks (like Spectre or Meltdown), though it is designed to be expanded in these directions. Future iterations aim to move from binary pass/fail results to a more nuanced quantitative security scoring system.

Summary updated

BibTeX
@misc{song2021comprehensive,
  author = {Wei Song and Jiameng Ying and Sihao Shen and Boya Li and Hao Ma and Peng Liu},
  title = {{A Comprehensive and Cross-Platform Test Suite for Memory Safety - Towards an Open Framework for Testing Processor Hardware Supported Security Extensions}},
  year = {2021},
  eprint = {2111.14072},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2111.14072},
}
Figure