An Empirical Study of Issues in GPU Kernel Libraries for LLM Systems

Yanjie Gao, Jiayi Bi, Haoxiang Lin, Ling FengarXiv 2026 · arXiv

TL;DR

This paper presents the first comprehensive empirical study of the bugs and failures lurking within the specialized GPU kernel libraries—like FlashAttention and Triton—that power modern Large Language Models (LLMs). By analyzing 300 high-quality GitHub issues, the authors reveal that over 60% of problems actually stem from the surrounding compiler and runtime environment rather than the kernel code itself, highlighting a critical need for better full-stack validation and debugging tools.

What Problem Does This Paper Address?

As LLMs scale, developers increasingly rely on highly optimized GPU kernel libraries to handle performance-critical operations like attention, General Matrix Multiplication (GEMM), and quantization. These libraries sit in a precarious spot: below high-level frameworks like PyTorch but above the raw hardware.

The challenge is that this layer is incredibly complex. A single kernel might be specialized for specific tensor shapes, data types (like FP8 or INT4), and GPU architectures. When things go wrong at this level, the failures are often catastrophic or, worse, silent. A bug might cause a GPU to hang, prevent a model from even being deployed, or silently corrupt numerical outputs—leading to "hallucinations" that are actually just math errors. Despite their importance, there hasn't been a systematic study of why these libraries fail until now.

Methods

The researchers conducted a manual, deep-dive analysis of 300 high-quality GitHub issue reports sampled from seven representative open-source libraries. They didn't just look at the initial bug reports; they examined the entire lifecycle of the issue, including developer discussions, reproduction scripts, diagnostic evidence, and the final code fixes.

To ensure the study was rigorous, two authors independently labeled the issues based on symptoms and root causes, achieving a high level of agreement (Cohen’s kappa of 0.87). They categorized the findings into a taxonomy of 6 symptom categories and 15 root-cause categories organized across three dimensions: the Kernel Library itself, the Compiler and Build System, and the Runtime Environment.

Datasets

The "dataset" for this study consists of the issue trackers from seven pivotal libraries in the LLM ecosystem. The authors filtered a raw collection of over 9,000 issues down to a candidate pool of 3,181 closed, failure-related reports, finally sampling 300 for detailed analysis.

Table II: Classification of common symptoms observed in the study.

The libraries included:

  • Attention/Operator focused: FlashAttention, FlashInfer, xFormers.
  • GEMM/Matrix focused: CUTLASS, DeepGEMM.
  • Compilation/Quantization focused: Triton, bitsandbytes.

Key Insights and Ideas

  • The "Pre-Execution" Trap: More than half of the observed symptoms (Deployment and Build failures) occur before a single line of kernel code even runs on the GPU. This suggests that "reliability" in this field is as much about managing complex software dependencies and compiler toolchains as it is about writing fast CUDA code.
  • The Danger of Specialization: To get maximum speed, kernels are often "specialized" for specific constraints (e.g., "this kernel only works if the head dimension is a multiple of 32"). These constraints are often implicit. When a user provides an "unconventional" shape, the library might select an incorrect kernel or fail to launch, leading to silent errors or crashes.
  • Silent Numerical Corruption: Incorrect functionality is a major headache. The study found cases where kernels produced finite but wildly inaccurate results because of subtle issues like inappropriate precision casting or "order-sensitive" reductions in low-precision math.
  • The "Stuck" Illusion: Build failures often look like "hangs." For instance, a missing dependency might cause a build to fall back to a non-parallel path that is so slow it appears the system has frozen.

Key Results

The study provides a clear quantitative map of the "danger zones" in GPU kernel development:

  • Symptoms: Deployment Failure (27.33%) and Build Failure (24.33%) are the most common user-facing issues, followed by Crashes (21.00%) and Incorrect Functionality (17.67%).
Table II: Classification of common symptoms observed in the study.
  • Root Causes: Software Incompatibility (25.33%) is the single largest root cause, highlighting the fragility of the CUDA/PyTorch/Driver stack. Within the kernel libraries themselves, API Misuse (8.67%) and Code Logic Errors (7.33%) are the primary culprits.
Table III: Taxonomy of root causes organized by dimension.

Limitations and Future Work

The authors acknowledge that their study is limited to open-source libraries and relies on the quality of public GitHub documentation. Proprietary kernels or issues that were never reported publicly are not captured.

Future Directions: 1. Compatibility Checkers: Tools that can pre-flight the entire stack (GPU, driver, CUDA version, ABI) before a build begins. 2. Contract-Aware Validation: Moving from implicit constraints to explicit "contracts" in code that validate tensor shapes and memory layouts before kernel launch. 3. Numerical Debugging Tools: Better tooling to track how precision is lost during intermediate stages of a GPU kernel's execution. 4. Record and Replay: Systems that can capture the exact state (metadata, IR, and inputs) that triggered a failure to make reproduction easier for maintainers.

Summary updated

Abstract

Behind the expanding use of large language models (LLMs) across application domains lies a growing dependence on specialized GPU kernel libraries. These libraries underpin the correctness and efficiency of training and inference across diverse GPU platforms, implementing attention, GEMM, quantization, and low-precision operators while managing tensor shapes, memory layouts, code generation, and hardware-specific dispatch. Problems at this layer can silently corrupt outputs, hang GPUs, prevent compilation, introduce deployment incompatibilities, or degrade performance, thereby reducing developer productivity and wasting expensive GPU resources. We present the first comprehensive empirical study of issues in GPU kernel libraries for LLM systems. We manually analyzed 300 high-quality GitHub issue reports, their discussions, and available fixes from seven representative libraries: FlashInfer, FlashAttention, Triton, CUTLASS, DeepGEMM, xFormers, and bitsandbytes. The analysis characterizes symptoms, root causes, fixes, and testing and debugging practices. Our main findings are: (1) 60.33% of the sampled issues originate in the Compiler and Build System or Runtime Environment dimensions, so validation must cover the deployment and execution stack, not just kernel code. (2) Kernel specialization imposes implicit correctness constraints on tensor shapes, metadata layouts, memory aliasing, and hardware resources. Violations can produce incorrect outputs or launch failures even for supported operations, highlighting the need for explicit contract validation and boundary-case testing. (3) Current testing and debugging practices are often insufficient for identifying low-precision numerical problems and hardware-specific execution failures. We outline research opportunities and tooling directions to support future LLM GPU kernel development.
BibTeX
@misc{gao2026empirical,
  author = {Yanjie Gao and Jiayi Bi and Haoxiang Lin and Ling Feng},
  title = {{An Empirical Study of Issues in GPU Kernel Libraries for LLM Systems}},
  year = {2026},
  eprint = {2611.09280},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.09280},
}
Figure