TL;DR
Modern multi-processor systems-on-chip (MPSoCs) are crucial for critical applications, but managing contention between tasks is a major hurdle. This paper dives deep into the Xilinx Zynq UltraScale+ MPSoC's hardware Quality of Service (QoS) mechanisms, providing the first comprehensive analysis of how these distributed features work together (and sometimes against each other) to control contention and ensure predictable performance.
What Problem Does This Paper Address?
As MPSoCs become more powerful and integrated into safety-critical systems like those in avionics and automotive, ensuring predictable task execution is paramount. A key challenge is "multicore contention," where multiple tasks running concurrently on different processing cores interfere with each other's access to shared resources like memory. This interference can lead to unpredictable timing behavior, making it difficult to guarantee that critical tasks will always meet their deadlines. While MPSoCs like the Xilinx Zynq UltraScale+ (ZUS+) offer hardware-based Quality of Service (QoS) mechanisms to manage this contention, these mechanisms are distributed across various IP blocks, and their individual and combined behavior is often poorly documented. Understanding and effectively configuring these distributed QoS features is essential for achieving the desired predictability, but this is a complex task due to the sheer number of QoS points and potential configurations.
Methods
The authors employ a multi-pronged approach:
- Qualitative Analysis: They meticulously analyze the technical documentation for various IP blocks (Arm NIC-400, Arm CCI-400, Synopsys uMCTL2) that implement QoS features within the ZUS+. This involves identifying individual QoS mechanisms and their functionalities.
- Empirical Characterization: Crucially, they go beyond documentation by conducting controlled experiments on a Xilinx ZCU102 board. This allows them to:
- Uncover undocumented QoS behaviors and implementation choices made by Xilinx.
- Identify incompatibilities and incongruities between different QoS mechanisms.
- Derive practical QoS configurations that can overcome limitations.
- QoS Domain Abstraction: To manage the complexity of distributed QoS, they introduce the concepts of "QoS domains" (groups of QoS-enabled devices operating under the same QoS abstraction) and "QoS domain mapping" (how priorities are related across these domains).
- Case Study: A realistic deployment scenario involving neural network kernels is used to demonstrate how their findings can be applied to configure QoS for achieving specific predictability goals for different software partitions.
Datasets
Not discussed in the paper. The authors use custom benchmarks and workloads generated using Xilinx's AXI Traffic Generators (ATG) to stress the memory system and evaluate QoS configurations.
Key Insights and Ideas
- The "Distributed Nightmare": The ZUS+ has a vast number of QoS points (over 30) spread across its interconnect and memory controller, leading to millions of potential configurations. Simply mastering individual QoS features isn't enough; their coordinated operation is key.
- Documentation Gaps are Real: Xilinx's documentation for the ZUS+ is surprisingly sparse regarding the specific implementation details of QoS features. The authors had to resort to empirical analysis to uncover critical information.
- Incompatibility is a Showstopper: Not all QoS mechanisms play nicely together. For instance, dynamic QoS features in the interconnect can conflict with the static QoS handling in the DDR controller, leading to unpredictable outcomes.
- Routing is Your Friend: A significant limitation is that all four A53 cores share a single port to the CCI. However, the authors cleverly exploit how memory addresses are mapped to different DDR controller ports to effectively provide different QoS levels to individual A53 cores.
- QoS Domains for Sanity: The introduction of QoS domains and mappings provides a structured way to think about and analyze the complex interactions between different QoS mechanisms.
Key Results
- First Comprehensive Analysis: The paper presents what is claimed to be the first qualitative and quantitative analysis of the distributed QoS mechanisms in the Xilinx Zynq UltraScale+ MPSoC.
- Identification of Incompatibilities: They identified several incompatible QoS features, particularly between dynamic QoS mechanisms and the DDR controller.
- Workarounds for Limitations: They demonstrated a method to provide different QoS levels to individual A53 cores by leveraging address mapping and routing, overcoming a key architectural limitation.
- QoS Malleability: The study shows that QoS configurations are "malleable," meaning they can effectively bias execution towards higher-priority tasks, even if it means other tasks experience more contention. This is crucial for QoS to be useful.
- Case Study Success: In their case study, they successfully identified QoS setups that met diverse timing constraints for different software partitions, demonstrating the practical applicability of their findings. For example, they showed that with specific QoS configurations, they could achieve a high ratio of accepted QoS setups (e.g., ~30% for tight constraints on RPU1, and even higher with relaxed constraints).
Limitations and Future Work
- Limited Documentation: The authors repeatedly highlight the lack of detailed documentation as a significant challenge, requiring extensive empirical work.
- Complexity of Analysis: The analysis of QoS mechanisms required significant effort from hardware experts, indicating the inherent complexity of the problem.
- Formalizing the Process: Future work aims to formalize a more generic process for orchestrating QoS features in other MPSoCs, including identifying QoS domains, mapping them, and finding compatible features.
- Automated Search: Developing advanced search algorithms is proposed to efficiently explore the vast QoS configuration space for more complex workloads and predictability constraints.
- Test-Driven Validation: The authors suggest aligning with avionics and automotive practices by formulating test designs to validate QoS setups against specific system behaviors.
Summary updated