A Lightweight and Portable Approach to Making Concurrent Failures Reproducible

Qingzhou Luo, Sai Zhang, Jianjun Zhao, Min HuFASE 2010 · OpenReview

Abstract

Concurrent programs often exhibit bugs due to unintended interferences among the concurrent threads. Such bugs are often hard to reproduce because they typically happen under very specific interleaving of the executing threads. Basically, it is very hard to fix a bug (or software failure) in concurrent programs without being able to reproduce it. In this paper, we present an approach, called ConCrash, that automatically and deterministically reproduces concurrent failures by recording logical thread schedule and generating unit tests. For a given bug (failure), ConCrash records the logical thread scheduling order and preserves object states in memory at runtime. Then, ConCrash reproduces the failure offline by simply using the saved information without the need for JVM-level or OS-level support. To reduce the runtime performance overhead, ConCrash employs a static data race detection technique to report potential possible race conditions, and only instruments such places. We implement the ConCrash approach in a prototype tool for Java and experimented on a number of multi-threaded Java benchmarks. As a result, we successfully reproduced a number of real concurrent bugs (e.g., deadlocks, data races and atomicity violation) within an acceptable overhead.

Introduction

Concurrent programs are notoriously difficult to debug because their behavior depends on the specific interleaving of threads. A bug might appear once in a million executions and then vanish when you try to attach a debugger—a phenomenon known as a "heisenbug." These failures typically fall into categories such as data races, deadlocks, and atomicity violations.

Examples of concurrent bug types: (i) data race, (ii) atomicity violation, and (iii) deadlock.

Existing solutions for "record and replay" typically require custom versions of the Java Virtual Machine (JVM) or specific Operating System support, limiting their portability. Others are restricted to single-threaded programs. ConCrash aims to bridge this gap by creating a tool that is both lightweight (low performance overhead) and portable (works on any standard JVM).

Methodology: The ConCrash Approach

ConCrash operates in three distinct stages to achieve deterministic reproduction without modifying the underlying infrastructure.

The ConCrash workflow, consisting of preprocessing, capture & replay, and offline analysis stages.
  1. Stage One: Static Pre-filtering. To avoid the massive overhead of monitoring every single instruction, ConCrash uses a static analysis tool called Chord to identify potential data race sites. It only instruments these "high-risk" areas and lock-related operations (acquire/release).
  2. Stage Two: Runtime Recording. Using the Soot framework, ConCrash performs bytecode-level instrumentation. It records a "logical thread schedule" based on Lamport clocks—capturing the order of critical events rather than absolute time. It also maintains a "shadow stack" that stores copies of method arguments and receiver objects.
  3. Stage Three: Offline Reproduction. When a crash occurs, ConCrash saves the schedule and object states to a small metadata file. It then automatically generates a suite of JUnit tests. These tests use a custom WrapperThread and Java Reflection to recreate the exact state of all threads and force them to execute in the recorded order, reliably triggering the crash for the developer.

Evaluation and Results

The authors evaluated ConCrash using six multi-threaded Java benchmarks, including real-world applications like the Apache FTP Server and web crawlers.

  • Successful Reproduction: ConCrash successfully reproduced all three major types of concurrent bugs (deadlocks, data races, and atomicity violations) across all test subjects.
  • Acceptable Overhead: Runtime performance overhead ranged from 15.3% to 66%. While significant, this is often acceptable for testing environments.
  • Trace Efficiency: Metadata files were remarkably small (max 45KB), facilitating easy bug reporting.
  • Effectiveness: In the hedc benchmark, a bug that appeared once in 300+ runs was reproduced 100% of the time with ConCrash.

Discussion and Conclusion

The key innovation of ConCrash is moving instrumentation to the bytecode level, ensuring portability across standard Java environments. By combining static analysis with dynamic recording, it achieves reproducibility with lower overhead than full-system logging. Furthermore, providing reproduction artifacts as standard JUnit tests allows developers to use familiar IDE debuggers to step through failing interleavings.

While effective, ConCrash currently has limitations regarding external non-determinism (e.g., network I/O) and its reliance on shallow object copying. Future work includes integrating dynamic program slicing to further reduce overhead and exploring more sophisticated object-capturing strategies.

Summary updated

Abstract

Concurrent programs often exhibit bugs due to unintended interferences among the concurrent threads. Such bugs are often hard to reproduce because they typically happen under very specific interleaving of the executing threads. Basically, it is very hard to fix a bug (or software failure) in concurrent programs without being able to reproduce it. In this paper, we present an approach, called ConCrash, that automatically and deterministically reproduces concurrent failures by recording logical thread schedule and generating unit tests. For a given bug (failure), ConCrash records the logical thread scheduling order and preserves object states in memory at runtime. Then, ConCrash reproduces the failure offline by simply using the saved information without the need for JVM-level or OS-level support. To reduce the runtime performance overhead, ConCrash employs a static data race detection technique to report potential possible race conditions, and only instruments such places. We implement the ConCrash approach in a prototype tool for Java and experimented on a number of multi-threaded Java benchmarks. As a result, we successfully reproduced a number of real concurrent bugs (e.g., deadlocks, data races and atomicity violation) within an acceptable overhead.
BibTeX
@inproceedings{luo2010lightweight,
  author = {Qingzhou Luo and Sai Zhang and Jianjun Zhao and Min Hu},
  title = {{A Lightweight and Portable Approach to Making Concurrent Failures Reproducible}},
  booktitle = {FASE},
  year = {2010},
  url = {https://openreview.net/forum?id=VUbCtVO7HT0},
}
Figure