TL;DR
This paper introduces Deux, an autonomic system for testing operating system upgrades that minimizes downtime and eliminates the need for specialized IT expertise or extensive regression suites. By leveraging a dual-layer virtual machine architecture with lightweight process checkpointing, Deux allows users to run applications on both the old and new OS versions simultaneously, using the old OS as a "pseudo-oracle" to detect discrepancies.
What Problem Does This Paper Address?
Operating system upgrades and patches, while necessary for new features and security, frequently introduce compatibility issues that can break existing applications, leading to system malfunctions or crashes. This risk makes users and organizations hesitant to adopt new OS versions promptly. Traditional beta testing and package management systems fall short because they cannot anticipate the vast array of user-specific application environments and configurations. Furthermore, these solutions often require significant IT expertise, dedicated hardware, and manual testing efforts, making them inaccessible for small businesses or individual users. The paper aims to bridge this gap by providing an automated, user-friendly solution that allows for seamless testing of OS upgrades within the user's existing environment.
Methods
The core of Deux's approach lies in its innovative use of a dual-layer virtual machine (VM) architecture combined with lightweight application process checkpointing and resume.
- Virtual Machine Environment: Deux utilizes two VMs. VM1 runs the current, stable operating system (OS1), and VM2 runs the new, upgraded operating system (OS2). This provides isolation and allows for parallel execution without requiring new hardware.
- Process Domain (POD) Architecture: Within each VM, applications run inside a POD, which is a group of processes with a virtualized view of the system. This POD architecture, based on Zap, is crucial because it allows for the migration and resumption of application processes across different OS versions, unlike full VM checkpointing which would include the entire OS.
- Autonomic and Simultaneous Execution: The system monitors user input on the OS1 instance (VM1) and automatically replicates it to the OS2 instance (VM2). This ensures that the same application is subjected to identical inputs on both operating systems.
- Pseudo-Oracle Testing: The OS1 instance acts as a "pseudo-oracle." Any output discrepancies between the applications running on OS1 and OS2 are logged. This allows Deux to identify potential defects introduced by the OS upgrade.
- Checkpoint and Restart: When a discrepancy is detected, or periodically, the system checkpoints the state of the application processes in both VMs. The checkpointed state from VM1 is then used to restart the application in VM2, ensuring that testing can continue from an identical state. This process is transparent to the end-user.
- Automated Role Switching: If testing reveals that the new OS version is stable and all discrepancies are deemed inconsequential, Deux can facilitate a seamless switchover, making OS2 the production OS with the application state already in place.
Datasets
Not discussed in the paper. The evaluation uses real-world applications like phpBB and Mozilla Firefox, but these are not presented as formal datasets.
Key Insights and Ideas
- Leveraging the Old OS as a Pseudo-Oracle: This is a clever way to automate defect detection without needing pre-defined test cases. The user's normal activity becomes the test suite.
- Lightweight POD Checkpointing for Cross-OS Migration: The ability to checkpoint and resume individual application processes (PODs) rather than entire VMs is key to enabling state synchronization and migration across different OS versions without the overhead of full VM cloning.
- Minimizing Downtime and IT Expertise: The system's design prioritizes making OS upgrade testing accessible to non-technical users, directly addressing a significant barrier to adoption.
- Potential for Both Defect Detection and Regression Identification: Deux can not only find new bugs introduced by an upgrade but also identify bugs from the old OS that have been fixed in the new one, providing a more comprehensive view of the upgrade's impact.
Key Results
The paper presents empirical evaluations using a LAMP server application (phpBB) and a standalone application (Mozilla Firefox).
- Defect Detection: Deux successfully identified a known defect in phpBB related to email functionality that only manifested with the new OS version (OS2). It also detected a defect in Mozilla Firefox caused by a conflict with a newer OS library (libnss3), leading to crashes.
- Fixed Defects: In one instance with Mozilla Firefox, Deux showed that the newer OS version (OS2) fixed a defect present in the older OS version (OS1) related to the mime-support package.
- Performance Metrics:
- Memory Usage: Deux added approximately 782MB of memory usage on top of the base VMs, which was deemed acceptable for a system with 4GB of RAM.
- CPU Usage: The CPU overhead for Deux itself (running two VMs and two PODs) was around 1.1% of the total CPU, with the applications running in PODs consuming up to 30.5% during peak activity.
- Checkpoint/Restart Times: Checkpointing and restarting times were consistently low, typically less than one second, and often in the milliseconds range, making them negligible to the end-user experience.
Limitations and Future Work
- Limitations:
- The system currently cannot address intentionally non-deterministic applications.
- The prototype implementation does not handle Read Only and Read/Write data of user application databases separately, which could lead to increased storage space requirements.
- Output discrepancy analysis for non-deterministic results currently requires human judgment.
- Future Work:
- Incorporating a copy-on-write database to better manage application databases.
- Developing semi-automated analysis tools to aid in checking application discrepancies, especially for non-deterministic outputs.
- Automating the construction of local regression test suites from normal user activities.
- Extending the technology to other operating systems like Windows.
Summary updated