Detection Software Content Failures using Dynamic Execution Information

Shiyi Kong, Minyan Lu, Bo Sun, Jun Ai, Shuguang Wang

TL;DR

Detecting "content failures"—where software stays running but produces incorrect output—is notoriously difficult compared to spotting crashes. This paper proposes a novel detection method using software runtime entropy, a metric that quantifies the "randomness" of function execution times and call patterns, to train machine learning models that identify silent failures with high accuracy.

What Problem Does This Paper Address?

Most software reliability research focuses on "hard" failures like crashes, hangs, or memory leaks (aging-related failures). However, the authors point out that content failures—where the system's output deviates from the intended function while the process remains active—actually account for the majority of real-world software issues.

Because the system doesn't "break" in a traditional sense, these errors are often imperceivable to standard monitoring tools. Traditional verification and validation (V&V) often miss these "resident defects." The challenge is to find a runtime indicator that can signal a content failure is occurring without needing to know the "correct" answer in advance (which would require a perfect oracle).

Methods

The researchers propose a pipeline that moves from raw execution data to a machine-learning-based verdict:

  1. Dynamic Binary Instrumentation: They record "execution traces"—sequences of function entries, exits, and timestamps—while the software is running.
  2. Software Runtime Entropy ($H$): This is the paper's core innovation. They calculate two types of entropy based on Shannon’s information theory:
  3. Execution Time Entropy ($H_A$): Measures the distribution of time spent in different functions.
  4. Function Call Entropy ($H_B$): Measures the frequency and patterns of how functions call one another.
  5. The final indicator is the average of these two: $H = \frac{1}{2}(H_A + H_B)$.
  6. Classification: These entropy measures are fed into a C4.5 Decision Tree algorithm. To handle the "imbalanced data" problem (where you usually have much more healthy data than failure data), they use SMOTE (Synthetic Minority Over-sampling Technique) to generate synthetic examples of failures for better training.

Datasets

The authors used three well-known open-source C projects from the Software-artifact Infrastructure Repository (SIR):

  • grep: A command-line text search utility.
  • flex: A fast lexical analyzer generator.
  • gzip: A file compression/decompression application.

Since real-world failure data is scarce, they used fault injection (seeding the code with intentional bugs) to create "failed" states and generate the necessary traces for training and testing.

Key Insights and Ideas

  • Entropy as a Health Monitor: The "aha moment" is that software execution has a "rhythm." When a bug is triggered, even if it doesn't cause a crash, it subtly disrupts the timing and flow of function calls. Entropy captures this disruption as a measurable shift in "information randomness."
  • Content Failures have Fingerprints: Even if the output is just a slightly wrong number, the path the CPU took to get there usually looks different from a successful run.
  • The Complexity Trade-off: The authors found that using a composite indicator (entropy) is better than using raw duration or call counts because it reduces the "feature space." This makes the machine learning models more generalized and less prone to overfitting on specific, irrelevant details.

Key Results

The models were evaluated using Precision, True Positive Rate (TPR), and F1-measure:

  • High Accuracy: For grep and gzip, the models achieved F1-measures consistently above 0.90, indicating very high reliability in detecting content failures.
  • The "Flex" Challenge: The flex dataset was more difficult. Because the original data was heavily skewed toward failed executions, the False Positive Rate (FPR) was higher (around 0.3-0.4). However, applying SMOTE helped bring the FPR down, showing that data balancing is critical for this approach.
  • Model Simplicity: Increasing the "Parameter M" (minimum instances per leaf in the decision tree) simplified the models. While this slightly reduced accuracy, it made the detection rules much clearer and easier to implement in a real-time environment.

Limitations and Future Work

  • Instrumentation Overhead: Collecting detailed function-level traces (especially timestamps) can slow down software. The authors acknowledge this but don't provide a detailed performance benchmark for the overhead.
  • Fault Injection vs. Reality: The study relies on "fault seeds." It remains to be seen how well these entropy patterns translate to "wild" bugs that emerge from complex, multi-threaded interactions in production environments.
  • False Alarm Reduction: The authors identify the need to further reduce false alarm rates, particularly in software types like flex where the execution patterns are naturally more varied. Future work will likely focus on refining the entropy calculation to be more robust against "noisy" but healthy execution variations.

Summary updated

Abstract

Modern software systems become more and more complex, which makes them difficult to test and validate. Detecting software partial failures in complex systems at runtime assist to handle software unintended behaviors, avoiding catastrophic software failures and improving software runtime availability. These detection techniques aim to find the manifestation of faults before they finally lead to unavoidable failures, thus supporting following runtime fault-tolerant techniques. We review the state-of-the-art articles and find that the content failures account for the majority of all kinds of software failures, but its detection methods are rarely studied. In this work, we propose a novel failure detection indicator based on the software runtime dynamic execution information for software content failures. The runtime information is recorded during software execution, then transformed to a measure named runtime entropy and finally fed into decision tree models. The machine-learning models are built to classify the intended and unintended behaviors of the objected software systems. A series of controlled experiments on several open-source projects are conducted to prove the feasibility of the method. We also evaluate the accuracy of machine-learning models built in this work.
BibTeX
@inproceedings{kong2021detection,
  author = {Shiyi Kong and Minyan Lu and Bo Sun and Jun Ai and Shuguang Wang},
  title = {{Detection Software Content Failures using Dynamic Execution Information}},
  booktitle = {QRS Companion},
  year = {2021},
  url = {https://openreview.net/forum?id=9qKerX9cXu},
}

Loading related content...

Figure