Automated Fault Injection for System Resilience

104 concepts79 papers1 proposal

Real-world hardware and software failures are rare, unpredictable, and catastrophic when they occur. Waiting for a natural failure to observe how a system responds is impractical. Automated Fault Injection (AFI) solves this by deliberately and systematically introducing artificial faults into a system under controlled conditions.

The core intuition behind why AFI works is boundary interception. Modern systems are built out of modular components separated by clear interfaces—hardware registers, network sockets, client drivers, or firmware APIs. By inserting a fault-generation mechanism directly at these boundaries, we can trick the system's error-handling logic into executing without needing to physically destroy hardware or permanently corrupt production environments.


Hardware and Microarchitectural Interception

At the lowest level, physical disturbances like radiation can cause transient bit-flips, often called "soft errors." To test how software handles these without expensive radiation chambers, researchers inject faults directly into the hardware description or execution pipeline.

For instance, Abhishek Tyagi demonstrates that Register Transfer Level (RTL) fault injection can simulate transient bit-flips directly within a processor's hardware description code. This approach provides high accuracy in tracking how hardware-level errors propagate to application-level failures compared to higher-level abstractions.

To achieve this without modifying physical hardware, virtualization and emulation are heavily utilized. Nathan A. DeBardeleben et al. developed a framework that modifies a virtual machine's instruction emulation engine to intercept specific opcodes during execution. By intervening at the emulator level, researchers can manipulate register values or operation results safely within a specific application's memory space, preventing the host operating system from crashing.

[Normal Execution]      Instruction Pointer -> VM Emulator -> Correct Register State
                                                    |
[Fault Injected]        Instruction Pointer -> VM Emulator -> Intercepted & Mutated State

Similarly, in hardware security, Alvise De Faveri Tron et al. showed that programmatically controlling a target system's Power Management Bus (PMBus) via low-cost microcontrollers allows for software-defined voltage manipulation (glitching) to induce instruction skips. For reconfigurable hardware, Bruno Zatt et al. highlighted that injecting faults into both FPGA configuration memory and internal user-defined memory blocks provides a realistic assessment of active workloads under environmental radiation.


Software and Distributed Systems Chaos

As we move up the stack to distributed systems, microservices, and decentralized networks, the boundaries shift from hardware registers to network protocols and API boundaries. Distributed systems are notoriously vulnerable to non-deterministic events like network delays, process crashes, or partial outages.

Stephan Krusche et al. demonstrate service-level fault injection by intercepting calls directly at the client-driver interface. This allows developers to simulate complex database failures, such as timeouts or "Byzantine faults" (corrupted data payloads), locally within their IDE. By automating simultaneous failures across multiple services, their approach achieves high code coverage of cascading outages that are difficult to coordinate manually.

In the context of decentralized applications (DApps), Lili Wei et al. explored this boundary in ÐArcher: detecting on-chain-off-chain synchronization bugs in decentralized applications. By modifying a blockchain client to systematically simulate non-deterministic transaction states (like dropped or reversed transactions), they exposed critical synchronization bugs between on-chain and off-chain components.

For large-scale enterprise databases, Artur Andrzejak et al. utilized a specialized testing mode that voluntarily triggers out-of-memory exceptions to verify graceful recovery from resource exhaustion.

Furthermore, automated fault injection serves as a foundational tool for benchmarking diagnostic utilities. Frameworks like RCAEval, developed by Hongyu Zhang et al., systematically introduce performance anomalies (e.g., resource exhaustion, request throttling) to generate labeled datasets. This allows researchers to rigorously evaluate automated Root Cause Analysis (RCA) algorithms against known ground truths.


Cyber-Physical and Model-Based Simulation

For safety-critical systems like autonomous underwater vehicles (AUVs), unmanned aerial vehicles (UAVs), and space habitats, injecting physical faults during operation could be catastrophic. Here, fault injection is integrated into high-fidelity simulations, digital twins, or model-checking environments.

Erik Frisk et al. developed a multi-category fault injection framework for engine systems that integrates sensor biases, actuator malfunctions, and mechanical leakages into a simulation. This allows diagnostic algorithms to be safely evaluated against complex, non-linear fault propagation.

In autonomous aviation, Azarakhsh Keipour et al. introduced controlled mechanical failures by modifying autopilot firmware to accept remote commands that override standard actuator behavior. This enabled the safe collection of real-world flight data for conditions like engine loss or stuck control surfaces.

[Firmware Boundary]     Autopilot Logic -> [ Override Hook ] -> Actuator Command
                                                 |
                                         Remote Fault Input

When verifying autonomous control logic before deployment, formal models are often mutated. Jonathan Ezekiel et al. used a specialized compiler to automatically modify a system model, introducing specific failure modes based on predefined timing rules to stress-test fault-tolerance logic. Similarly, Ran Wei et al. automated safety analysis by injecting virtual failure modes into a space launch vehicle's digital twin, correlating real-time sensor data with reliability models to map physical anomalies back to engineering requirements.


Why It Works: The Power of Boundary Interception

The unifying reason automated fault injection works across all these domains is its ability to decouple fault generation from system logic via boundary interception. Whether it is a virtual machine intercepting an opcode, a client driver mutating a database response, or a simulation framework overriding an actuator command, the target system cannot distinguish between a real physical failure and a programmatically induced one.

By synchronizing these injections with peak system activity—as originally pioneered by Timothy Tsai in the development of FTAPE—AFI forces dormant error-handling code paths to execute under high stress. This systematic pressure exposure reveals hidden vulnerabilities long before they manifest in production or flight.

Key Foundational Work

FTAPE: A fault injection tool to measure fault tolerance (1994) — contributed 3 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_automated_fault_injection_d2b1614a,
  title = {automated fault injection},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/automated-fault-injection/rdr_d2b1614a4237450cb0f7a3228744c136},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_automated_fault_injection_d2b1614a_v1,
  title = {automated fault injection},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/automated-fault-injection/rdr_d2b1614a4237450cb0f7a3228744c136/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake