Coverage evaluation of post-silicon validation tests with virtual prototypes

Kai Cong, Lei Li, Zhenkun Yang, Fei Xie2014 · DOI

TL;DR

This paper introduces an "online-capture, offline-replay" method to evaluate the effectiveness of post-silicon validation tests before the actual silicon chip is available, using virtual prototypes. This approach allows engineers to get early feedback on test quality and identify potential issues, ultimately saving time and cost in the hardware development cycle.

What Problem Does This Paper Address?

Developing and validating complex hardware, like network adapters, is a time-consuming and expensive process. A critical phase is "post-silicon validation," where tests are run on the actual silicon chip after it's manufactured. However, this phase often starts too late, leading to delays in debugging and fixing issues. High-quality tests need to be ready before the silicon is available to speed up this process. The challenge is: how do you accurately measure the quality and coverage of these tests when you don't have the physical hardware to run them on? Traditional methods like RTL emulation are slow and require a complete RTL design. This paper aims to bridge this gap by enabling early, reliable test coverage evaluation using virtual prototypes.

Methods

The core of the proposed solution is an "online-capture, offline-replay" approach.

  1. Online Capture: When a test suite runs on a virtual prototype (a software model of the hardware) within a virtual platform, the system captures minimal but crucial run-time data. This includes the initial state of the virtual device and a sequence of events (device requests and environment inputs) that occur during the test execution.
  2. Offline Replay: This captured data (initial state and event sequence) is then used to replay the test execution offline on the same virtual prototype. This replay is done in a controlled environment, allowing for detailed analysis and computation of test coverage metrics.

This method is designed to be lightweight during the capture phase, minimizing overhead on the virtual platform. The offline replay provides flexibility, enabling engineers to analyze the execution in detail, generate various coverage reports, and even debug step-by-step.

To ensure the accuracy of coverage estimates for the eventual silicon device, the approach is extended to include a post-silicon conformance checking step. After the silicon device is available, the same test suite is run on it. The captured silicon device states and event sequences are then used to replay the execution on the virtual prototype. This allows for a direct comparison of coverage and identification of any inconsistencies between the virtual and silicon device behaviors.

Datasets

The paper evaluates its approach using virtual prototypes of five popular network adapters: Intel E1000, Broadcom Tigon3, Intel EEPro100, AMD RTL8139, and Realtek PCNet. The test suites used include common network testing programs like ifconfig and ethtool. No new benchmarks were introduced; instead, existing network adapter models and standard testing tools were utilized.

Key Insights and Ideas

  • Virtual Prototypes as a Bridge: The paper highlights the significant potential of virtual prototypes, with their inherent observability and traceability, to serve as a reliable platform for early test coverage evaluation, something difficult to achieve with black-box silicon.
  • Minimal Data Capture is Key: The "online-capture, offline-replay" strategy is crucial. Capturing only the initial state and event sequence minimizes runtime overhead and ensures that the captured data is sufficient for detailed offline analysis, avoiding the need to decide all necessary information upfront.
  • Conformance is Crucial for Fidelity: The extension to post-silicon conformance checking is a vital step. It acknowledges that virtual prototypes aren't perfect replicas and that comparing their coverage results with actual silicon behavior, after fixing discrepancies, builds confidence in the early estimates.
  • Hardware-Specific Metrics Matter: Beyond standard software coverage metrics (function, statement, block, branch), the paper introduces and utilizes hardware-specific metrics like register coverage and transaction coverage. This is essential for understanding how well tests exercise the actual hardware functionalities.

Key Results

  • The proposed approach was applied to five network adapter virtual prototypes, and the results showed that a suite of common tests achieved high functional coverage.
  • The online capture mechanism introduced minimal overhead (e.g., ~3.6% for E1000).
  • Offline replay was efficient, taking only a few minutes to process tens of thousands of events with modest memory usage.
  • Post-silicon conformance checking identified a small number of inconsistencies (e.g., 7 for E1000, 6 for Tigon3) which were fixed by modifying a small number of lines of code in the virtual devices.
  • After fixing inconsistencies, the coverage reports from the virtual devices closely matched those from the silicon devices, validating the accuracy of the early coverage estimates.

Limitations and Future Work

  • Accuracy of Early Estimates: While the approach aims for high fidelity, the paper acknowledges that events issued to virtual devices might differ from those issued to silicon devices for the same tests. The conformance checking step helps mitigate this, but it's an ongoing challenge to ensure the virtual coverage is a "good approximation" of silicon coverage.
  • Transaction Differences: The paper notes that some transaction coverage differences can occur due to the speed disparity between physical machines and virtual platforms. While deemed "acceptable," this highlights a potential area for refinement.
  • Future Work: The authors plan to:
  • Research new coverage metrics better suited for evaluating hardware coverage.
  • Investigate how to use coverage results to measure validation completeness and guide test generation more effectively.

Summary updated

Abstract

High-quality tests for post-silicon validation should be ready before a silicon device becomes available in order to save time spent on preparing, debugging and fixing tests after the device is available. Test coverage is an important metric for evaluating the quality and readiness of post-silicon tests. We propose an online-capture offline-replay approach to coverage evaluation of post-silicon validation tests with virtual prototypes for estimating silicon device test coverage. We first capture necessary data from a concrete execution of the virtual prototype within a virtual platform under a given test, and then compute the test coverage by efficiently replaying this execution offline on the virtual prototype itself. Our approach provides early feedback on quality of post-silicon validation tests before silicon is ready. To ensure fidelity of early coverage evaluation, our approach have been further extended to support coverage evaluation and conformance checking in the post-silicon stage. We have applied our approach to evaluate a suite of common tests on virtual prototypes of five network adapters. Our approach was able to reliably estimate that this suite achieves high functional coverage on all five silicon devices.
BibTeX
@inproceedings{cong2014coverage,
  author = {Kai Cong and Lei Li and Zhenkun Yang and Fei Xie},
  title = {{Coverage evaluation of post-silicon validation tests with virtual prototypes}},
  booktitle = {Design,},
  year = {2014},
  doi = {10.7873/date2014.331},
}
Figure