Detection Software Content Failures Using Dynamic Execution Information

Shiyi Kong, Minyan Lu, Jun Ai, Shuguang Wang

Abstract

Modern software systems have become too complex to be fully tested and validated. Detecting software partial failures in complex systems at runtime assists in handling unintended behaviors, avoiding catastrophic failures, and improving runtime availability. These detection techniques aim to find the manifestation of faults before they lead to unavoidable failures, supporting subsequent runtime fault-tolerant techniques. We review state-of-the-art articles and find that content failures account for the majority of software failures, yet their detection methods are rarely studied. In this work, we propose a novel failure detection indicator based on software runtime dynamic execution information for software content failures. The runtime information is recorded during execution, transformed into a measure named runtime entropy, and fed into machine learning models. These models are built to classify the intended and unintended behaviors of the software systems. A series of controlled experiments on several open-source projects prove the feasibility of the method and evaluate the accuracy of the resulting machine learning models.

Introduction

Most software reliability research focuses on "hard" failures like crashes, hangs, or memory leaks. However, content failures—where the system's output deviates from the intended function while the process remains active—actually account for the majority of real-world software issues. Because the system does not "break" in a traditional sense, these errors are often imperceivable to standard monitoring tools. Traditional verification and validation (V&V) often miss these "resident defects."

Taxonomy of failure domains, highlighting content failures as a specific subset where timing may be correct but output is erroneous.

The challenge lies in finding a runtime indicator that can signal a content failure is occurring without needing a perfect oracle to know the "correct" answer in advance. This paper proposes using software runtime entropy to quantify the "randomness" of execution patterns as a proxy for system health.

Methodology

The proposed detection pipeline moves from raw execution data to a machine-learning-based verdict through several stages:

  1. Dynamic Binary Instrumentation: The system records "execution traces"—sequences of function entries, exits, and timestamps—while the software is running.
  2. Software Runtime Entropy ($H$): This is the core innovation. Two types of entropy are calculated based on Shannon’s information theory:
  3. Execution Time Entropy ($H_A$): Measures the distribution of time spent in different functions.
  4. Function Call Entropy ($H_B$): Measures the frequency and patterns of how functions call one another.
  5. The final indicator is the average: $H = \frac{1}{2}(H_A + H_B)$.
  6. Classification: These entropy measures are fed into a C4.5 Decision Tree algorithm. To handle the "imbalanced data" problem (where healthy data typically outweighs failure data), the SMOTE (Synthetic Minority Over-sampling Technique) is used to generate synthetic failure examples for training.
The overall architecture of the proposed failure detection method, from instrumentation to model evaluation.

Experimental Setup

The authors evaluated the method using three open-source C projects from the Software-artifact Infrastructure Repository (SIR): grep (text search), flex (lexical analyzer), and gzip (compression). Since real-world failure data is scarce, fault injection was used to seed the code with intentional bugs, creating the "failed" states necessary to generate traces for training and testing.

Key Results and Insights

  • Entropy as a Health Monitor: Software execution has a natural "rhythm." When a bug is triggered, even if it doesn't cause a crash, it subtly disrupts the timing and flow of function calls. Entropy captures this disruption as a measurable shift in information randomness.
  • High Detection Accuracy: For grep and gzip, the models achieved F1-measures consistently above 0.90, indicating high reliability in detecting content failures.
  • The "Flex" Challenge: The flex dataset proved more difficult due to skewed execution patterns. However, applying SMOTE helped bring the False Positive Rate (FPR) down, demonstrating that data balancing is critical for this approach.
  • Model Simplicity: Increasing the minimum instances per leaf in the decision tree simplified the models. While this slightly reduced accuracy, it made the detection rules clearer and easier to implement in real-time environments.

Conclusion

This research demonstrates that software runtime entropy is a viable indicator for detecting silent content failures. By capturing the subtle "fingerprints" left by faults in execution traces, machine learning models can identify unintended behaviors with high precision. Future work will focus on reducing instrumentation overhead and further refining entropy calculations to minimize false alarms in highly varied execution environments.

Summary updated

Abstract

Modern software systems become too complex to be tested and validated. Detecting software partial failures in complex systems at runtime assist to handle software unintended behaviors, avoiding catastrophic software failures and improving software runtime availability. These detection techniques aim to find the manifestation of faults before they finally lead to unavoidable failures, thus supporting following runtime fault tolerant techniques. We review the state of the art articles and find that the content failures account for the majority of all kinds of software failures, but its detection methods are rarely studied. In this work, we propose a novel failure detection indicator based on the software runtime dynamic execution information for software content failures. The runtime information is recorded during software execution, then transformed to a measure named runtime entropy and finally fed into machine learning models. The machine learning models are built to classify the intended and unintended behaviors of the objected software systems. A series of controlled experiments on several open source projects are conducted to prove the feasibility of the method. We also evaluate the accuracy of machine learning models built in this work.
BibTeX
@inproceedings{kong2021detection,
  author = {Shiyi Kong and Minyan Lu and Jun Ai and Shuguang Wang},
  title = {{Detection Software Content Failures Using Dynamic Execution Information}},
  booktitle = {QRS Companion},
  year = {2021},
  url = {https://openreview.net/forum?id=9qKerX9cXu},
}
Figure