Abstract
TorchTraceAP introduces the first benchmark dataset specifically designed for detecting "performance anti-patterns"—such as redundant memory copies or CPU-GPU synchronization bottlenecks—within PyTorch execution traces. To address the challenge of manual trace inspection, the authors propose a hybrid, two-stage framework. This system utilizes a lightweight, hierarchical machine learning model to pinpoint problematic trace segments and a Large Language Model (LLM) to provide fine-grained diagnostic feedback and optimization suggestions.
Introduction
Developing high-performing Computer Vision (CV) models requires more than just high accuracy; execution efficiency is paramount for real-world deployment. However, identifying performance bottlenecks, or anti-patterns, typically necessitates "full-stack" expertise spanning ML modeling, computer architecture, and systems profiling.
Tools like the PyTorch Profiler generate detailed execution traces, but these logs are massive, often containing thousands of events for a single model pass. Manually inspecting these traces is a significant challenge. Existing automated solutions are often limited: rule-based systems struggle to generalize across hardware, and LLMs cannot process the massive context of a full execution trace due to token limits and high costs. TorchTraceAP aims to bridge this gap by providing a standardized dataset and an efficient automated detection pipeline.
The TorchTraceAP Dataset
The paper introduces TorchTraceAP, a comprehensive benchmark designed to train and evaluate performance diagnostic tools. The dataset includes over 600 PyTorch traces collected from diverse CV tasks, including classification (ResNet, ViT), detection (YOLO, DETR), segmentation (Segformer), and generation (Stable Diffusion).
Key features of the dataset include:
- Multi-Hardware Coverage: Traces recorded on NVIDIA RTX 4090, A6000, A100, and H100 GPUs.
- Granular Labels: Includes global "optimized/unoptimized" trace labels and fine-grained window-level annotations for eight specific anti-pattern types, such as "High Memcpy between Host and Device" and "Torch Graph Breaks."
- Hierarchical Representation: The dataset captures the relationship between CPU operations and the GPU kernels they launch, which is critical for identifying idle time and synchronization issues.
Proposed Framework
The authors propose a hierarchical, two-stage approach to transform raw trace data into actionable insights. This framework effectively "compresses" the system's reasoning task, allowing LLMs to act as expert consultants on small, relevant data chunks.
Stage 1: Lightweight Detection
The first stage acts as a filter. It uses a custom encoder to fuse textual operation names (via RoBERTa) with numerical data like launch delays and durations. Events are grouped into temporal windows and processed by a Transformer-based "Window Encoder" to capture intra-window relationships. A subsequent "Window-to-Window" detector uses dilated convolutions and self-attention to identify patterns across the entire timeline. The model is trained using Multiple Instance Learning (MIL), allowing it to learn from global trace labels without requiring exhaustive millisecond-level annotations.
Stage 2: LLM Analysis
Once the ML model flags a suspicious window, that specific segment is sent to an LLM (e.g., GPT-4). The LLM classifies the specific anti-pattern type and suggests fixes, bypassing context-length limitations while providing high-level diagnostic expertise.
Key Results
The proposed framework demonstrates significant improvements over baseline methods:
- Detection Performance: The ML model outperformed rule-based statistical methods by 15% in detecting anti-pattern regions.
- Generalization: The model successfully detected bottlenecks in image generation and depth estimation models, even when trained only on classification and detection traces.
- LLM Accuracy: Once a bottleneck was localized, the LLM achieved 75% accuracy in correctly classifying the specific anti-pattern type.
- Ablation Studies: Results confirmed that modeling the sequence and hierarchical relationship of events is more critical for detection than simply analyzing individual event durations.
Conclusion and Future Work
TorchTraceAP provides a vital resource for the ML systems community. While the current dataset is focused on Computer Vision, future work aims to extend this to Large Language Model training traces, which involve complex communication patterns like 3D parallelism. Additionally, integrating this detection pipeline directly into training loops could enable real-time performance regression alerts for engineers.
Summary updated