Abstract Domain Selection with Offline Reinforcement Learning

Qianshan Yu, Yang Liu, Feifan Xu, Can Yang, Zhijie Xu, Fei HeZenodo 2022 · DOI · OpenAlex

Abstract

This document shows how to checkout the results and reproduce the experiments. In addition to the reproduction instruction, we include all experimental results and the source code of our tools, which support the claims in our paper. We provide a virtual machine in <code>ova</code> format (md5:356d42a9684c23cca96fa938f2536b6d). When you finish downloading the VM, please import it to VMWare or Virtual Box (or other softwares that support <code>ova</code> format). Note that <strong>10GB</strong> of memory and <strong>4 CPU cores</strong> are the minimal requirements. The VM's username and password are <strong>domainsel</strong> and <strong>123456</strong>, respectively. SSH login is also supported. Reproduce 1. Experiment of Section 5.2 Effectiveness Start the experiment on the <strong>challenge</strong> set (our approach, <em>RLSel</em>): <pre><code class="language-shell">bash ~/RL_DomainSel/code/scripts/start_exp-5.2.1.sh </code></pre> At the end, enter <code>~/RL_DomainSel/code/cpachecker-refsel/</code> directory, and run the command <code>table-generator results/...bz2</code> shown on the terminal. The result table (in <code>csv</code> format) will be generated (Omit the <code>INFO: Missing property for task ...</code> information). Check the result (<code>csv</code> format) and log files (archived in <code>zip</code> format) in the <code>~/RL_DomainSel/code/cpachecker-refsel/results</code> directory. The results will vary with CPU performance. For example, I reproduced the experiment on a laptop with an Intel i5 CPU and the slowest 9 examples will be timeout. Efficiency Start the experiment on the <strong>ordinary</strong> set (baseline, <em>RefSel</em>): <pre><code class="language-shell">bash ~/RL_DomainSel/code/scripts/start_exp-5.2.2.sh </code></pre> At the end, run the <code>table-generator ...</code> command appeared on the terminal, and check the result and log files in the <code>~/RL_DomainSel/code/cpachecker-refsel/results</code> directory. Suppose the result file is <code>baseline-va-pa.results.cpa-refsel19.va-pa.csv</code>. Start the experiment on the <strong>ordinary</strong> set (our approach, <em>RLSel</em>): <pre><code class="language-shell">bash ~/RL_DomainSel/code/scripts/start_exp-5.2.3.sh </code></pre> At the end, run the <code>table-generator ...</code> command appeared on the terminal, and check the result and log files in the <code>~/RL_DomainSel/code/cpachecker-refsel/results</code> directory. Suppose the result file is <code>test-va-pa.results.cpa-refsel19.va-pa.csv</code>. Now run <code>python filter.py results/baseline-va-pa.results.cpa-refsel19.va-pa.csv results/test-va-pa.results.cpa-refsel19.va-pa.csv</code> in <code>~/RL_DomainSel/code/cpachecker-refsel</code> directory. The statistics for each category will be calculated automatically and saved in the <code>~/RL_DomainSel/code/cpachecker-refsel/result_csv</code> directory. Note that the speedup may not be the same as in Table 2 in the paper because of different CPU performance. We even have one timeout case in baseline when running on a laptop with an Intel i5 CPU. 2. Experiment of Section 5.3 Start the experiment on the <strong>ordinary</strong> set (baseline, <em>RefSel<sup>*</sup></em>): <pre><code class="language-shell">bash ~/RL_DomainSel/code/scripts/start_exp-5.3.1.sh </code></pre> At the end, run the <code>table-generator ...</code> command appearing on the terminal, and check the result and log files in the <code>~/RL_DomainSel/code/cpachecker-refsel/results</code> directory. Start the experiment on the <strong>ordinary</strong> set (our approach, <em>RLSel<sup>*</sup></em>): <em>Before running the script</em> <code>start_exp-5.3.2.sh</code>, <em>you need to comment the first line and uncomment the second line in file</em> <code>~/RL_DomainSel/code/domainsel/src/config.py</code> (<em>make the variable</em> <code>mode</code> <em>has the value</em> <code>'oct_pa_04_no-refsel-pa'</code>). <pre><code class="language-shell">bash ~/RL_DomainSel/code/scripts/start_exp-5.3.2.sh </code></pre> At the end, run the <code>table-generator ...</code> command appearing on the terminal, and check the result and log files in the <code>~/RL_DomainSel/code/cpachecker-refsel/results</code> directory. Directory information The working directory contains all experimental results (including logs and tables), the sets of verification tasks and source code of our tool. The following (relevant) files are in this directory: <pre><code>~/RL_DomainSel/ code/ The source code of neural network and our tool: - cpachecker-refsel/: The directory of our tool. - domainsel/: Include the network model and source code of RL network. - scripts/: Include startup scripts for all experiments. README.md This document. results/ The experimental results, including all log files and raw tables that correspond to the results in the paper. result_csv/ The result tables for each category generated by `filter.py` script. sets/ The lists of verification tasks of SV-Comp'19. </code></pre> (Note: You can modify the amount of memory and CPU required for the experiment as follows: Modify the line 14 of files <code>~/RL_DomainSel/code/cpachecker-refsel/test-*.xml</code> and <code>~/RL_DomainSel/code/cpachecker-refsel/baseline-*.xml</code> to reduce the required memory and CPU cores (<code>memlimit</code> for memory and <code>cpuCores</code> for CPU). Note that after lowering the resources, the experimental results may differ more from those in the paper.)
BibTeX
@inproceedings{yu2022abstract,
  author = {Qianshan Yu and Yang Liu and Feifan Xu and Can Yang and Zhijie Xu and Fei He},
  title = {{Abstract Domain Selection with Offline Reinforcement Learning}},
  booktitle = {Zenodo},
  year = {2022},
  doi = {10.5281/zenodo.7084867},
}

Loading related content...

Figure