Interference-free Operating System: A 6 Years' Experience in Mitigating Cross-Core Interference in Linux

Zhaomeng Deng, Ziqi Zhang, Ding Li, Yao Guo, Yunfeng Ye, Yuxin Ren, Ning Jia, Xinwei Hu

TL;DR

This paper details six years of industrial experience in identifying and fixing cross-core performance interference issues within the Linux kernel. By systematically addressing these problems across task management, resource management, and concurrency, the authors significantly improve real-time system predictability and schedulability, with their fixes now integrated into mainstream Linux.

What Problem Does This Paper Address?

Modern real-time systems heavily rely on multi-core processors for increased computing power. However, these systems demand strict timing guarantees, which are threatened by "cross-core performance interference" – where activity on one core unexpectedly delays tasks on another. While real-time operating systems (RTOS) aim for isolation, the Linux kernel itself, despite its widespread use, has become a major source of such interference. Existing research often focuses on isolated interference sources or user-level tasks, failing to provide a systematic approach to tackle the pervasive, complex, and often subtle interference issues originating from within the OS itself. This gap poses a significant risk to the time safety of latency-critical applications.

Methods

The authors' approach is rooted in extensive, hands-on industrial experience over six years. They systematically identified and fixed dozens of interference bugs across various Linux subsystems. Their methodology involved:

  • Categorization: Grouping interference issues into three main areas: task scheduling and placement, resource management and sharing, and concurrency and coordination.
  • Root Cause Analysis: Deeply investigating the underlying mechanisms in the Linux kernel that lead to these interferences.
  • Patch Development: Creating specific fixes for identified bugs, many of which have been merged into the mainline Linux kernel.
  • Systematic Lessons Learned: Distilling their experience into actionable advice for developers, focusing on unified isolation mechanisms, clear resource indicators, isolation-friendly synchronization, and verifiable programming practices.
  • Evaluation: Rigorously testing their solutions using micro-benchmarks (cyclictest, oslat) and real-world systems (Core Flight System, Robot Operating System 2).

Datasets

The paper does not introduce new datasets. Instead, it uses standard benchmarks and real-world applications for evaluation:

  • Micro-benchmarks: cyclictest and oslat are used to measure worst-case latency and jitter.
  • Real-world Applications:
    • Core Flight System (CFS): A middleware for satellite missions.
  • Robot Operating System 2 (ROS2): A framework for robotics development.
  • Interference Workloads: A set of benchmarks (listed in Table IV) designed to stress various parts of the system (task scheduling, timer management, I/O, CPU computation, kernel execution) without directly interacting with isolated real-time tasks.

Key Insights and Ideas

  • Linux as a Primary Interference Source: Contrary to the focus on external interference, the paper highlights that the Linux kernel itself is a significant and often overlooked source of timing unpredictability in real-time systems.
  • Systematic Approach is Crucial: Tackling cross-core interference requires a holistic view, addressing task management, resource management, and concurrency, rather than isolated fixes.
  • Complexity Breeds Interference: The intricate, evolved nature of the Linux kernel, with its numerous subsystems and independent development, creates a fertile ground for subtle, hard-to-detect interference issues.
  • Unified Mechanisms are Lacking: Linux lacks a cohesive strategy for managing core isolation, relying on a multitude of ad-hoc mechanisms that are difficult to manage and prone to inconsistencies.
  • Resource Indicators are Key: The absence of clear indicators for shared resource usage by isolated vs. non-isolated cores leads developers to inadvertently introduce interference.
  • Production Experience is Invaluable: The paper emphasizes the importance of industry practice in uncovering and fixing these deep-seated issues, which academic research alone might miss.

Key Results

The authors' work has led to significant improvements in Linux's real-time capabilities:

  • Reduced Jitter: Enhancements reduced worst-case jitter by a factor of 8.7x compared to vanilla Linux.
  • Improved Schedulability: Achieved a maximum 11.5x improvement in system schedulability.
  • Reduced Worst-Case Latency:
    • For cyclictest, openEuler (their modified Linux) achieved 8.7x lower worst-case latency than vanilla Linux and 4.0x lower than RT-Linux.
  • For Core Flight System (CFS), they achieved 2.1x and 1.6x reduction in worst-case response time compared to vanilla Linux and RT-Linux, respectively.
  • For ROS2, they achieved 1.64x reduction in maximum latency compared to RT-Linux in multi-executor scenarios, preventing packet loss.
  • Merged Fixes: Most of their modifications have been merged into the mainline Linux kernel (since version 4.19) and are available in commercial distributions.

Limitations and Future Work

  • Impossibility of Complete Fix: The authors acknowledge that completely fixing all cross-core interference bugs in Linux is impossible due to the lack of a well-designed, unified isolation enforcement mechanism.
  • Ad-hoc Fixes: Many of their fixes were developed in an "ad-hoc" manner to avoid intrusive modifications and ensure stable production environments.
  • Academic Involvement: They suggest increased academic participation to provide theoretical foundations and verification for real-time system design in Linux.
  • Unified Design: They advocate for designing future operating systems with unified isolation mechanisms, clear resource indicators, and isolation-friendly synchronization primitives.
  • Formal Verification: They propose making isolation interfaces and mechanisms more amenable to formal verification and automated program analysis.
  • Production-Driven Development: The paper notes that the community's focus on specific product needs can sometimes lead to incomplete or less general solutions, highlighting the need for better integration of academic research.

Summary updated

Abstract

Real-time operating systems employ spatial and temporal isolation to guarantee predictability and schedulability of real-time systems on multi-core processors. Any unbounded and uncontrolled cross-core performance interference poses a significant threat to system time safety. However, the current Linux kernel has a number of interference issues and represents a primary source of interference. Unfortunately, existing research does not systematically and deeply explore the cross-core performance interference issue within the OS itself. This paper presents our industry practice for mitigating crosscore performance interference in Linux over the past 6 years. We have fixed dozens of interference issues in different Linux subsystems. Compared to the version without our improvements, our enhancements reduce the worst-case jitter by a factor of 8.7, resulting in a maximum $11.5 x$ improvement over system schedulability. For the worst-case latency in the Core Flight System and the Robot Operating System 2, we achieve a 1.6x and $1.64 x$ reduction over RT-Linux. Based on our development experience, we summarize the lessons we learned and offer our suggestions to system developers for systematically eliminating cross-core interference from the following aspects: task management, resource management, and concurrency management. Most of our modifications have been merged into Linux upstream and released in commercial distributions.
BibTeX
@inproceedings{deng2024interference,
  author = {Zhaomeng Deng and Ziqi Zhang and Ding Li and Yao Guo and Yunfeng Ye and Yuxin Ren and Ning Jia and Xinwei Hu},
  title = {{Interference-free Operating System: A 6 Years' Experience in Mitigating Cross-Core Interference in Linux}},
  booktitle = {RTSS},
  year = {2024},
  url = {https://openreview.net/forum?id=rG7zAJgMVl},
}
Figure