TL;DR
This paper introduces PerfSeer, a novel deep learning model performance predictor that uses a graph-based representation to capture model topology and features. By leveraging Graph Neural Networks (GNNs) with novel aggregation and boosting techniques, PerfSeer achieves state-of-the-art accuracy in predicting execution time and resource utilization, outperforming existing methods.
What Problem Does This Paper Address?
Predicting the performance of deep learning (DL) models, such as their execution time and resource usage, is critical for various applications like Neural Architecture Search (NAS) and DL cluster scheduling. These technologies rely on accurate performance metrics to design efficient models, optimize resource allocation, and reduce job completion times. However, existing performance predictors often struggle with comprehensive model representation, leading to suboptimal accuracy. They either treat models as monolithic entities with global features or break them down into parts without fully capturing the intricate relationships and dependencies within the model's computational graph. This gap in accurate and comprehensive performance prediction hinders the advancement of DL systems.
Methods
PerfSeer tackles this problem in two main stages: representation and prediction.
1. Representation (Graph Extractor): PerfSeer first converts a DL model into a "performance graph" (PerfGraph) using a Graph Extractor. This graph goes beyond just the model's topology. It incorporates:
- Node Features: Representing individual operations (e.g., convolution, pooling) with details like hyperparameters, computation (FLOPs), memory access (MAC), arithmetic intensity, and proportions of these metrics relative to the whole model.
- Edge Features: Describing the data flow between operations, including tensor size and shape.
- Global Features: Capturing overall model characteristics like the number of nodes/edges, graph density, and aggregate statistics of node-level computation and memory access.
A key innovation here is the construction of several unique and effective features (e.g., arithmetic intensity, computation/memory access proportions) that were not previously used.
2. Prediction (SeerNet and SeerNet-Multi): The PerfGraph is then fed into a novel GNN-based prediction model called SeerNet. SeerNet is inspired by graph-based relational learning and features:
- SeerBlock: This core component allows each feature (node, edge, global) to update its representation by incorporating information from its own features and aggregated information from other features.
- Synergistic Max-Mean Aggregation (SynMM): A custom aggregation function that combines max and mean pooling to capture both prominent and general node features, leading to a more robust representation.
- Global-Node Perspective Boost (GNPB): A technique that introduces a "global node" to enrich the graph embedding with a broader perspective, enabling complementary learning between node and global features.
For predicting multiple performance metrics simultaneously, SeerNet-Multi is introduced. It extends SeerNet with multiple prediction heads and crucially incorporates Projecting Conflicting Gradients (PCGrad) to mitigate gradient conflicts that arise when training for multiple objectives, thus maintaining high accuracy.
Datasets
The paper introduces a new dataset containing performance metrics for over 53,000 model configurations. This dataset covers:
- Metrics: Execution time, memory usage, and Streaming Multiprocessor (SM) utilization.
- Scenarios: Both training and inference.
- Architectures: A wide range including VGG, GoogLeNet, ResNe(X)t, MobileNet, and DenseNet.
- Scale: Models with FLOPs ranging from 49 million to 22 trillion.
This dataset was collected on Nvidia GeForce RTX 3090 GPUs.
Key Insights and Ideas
- Graph Representation is Key: The paper strongly advocates for representing DL models as graphs that capture not just topology but also rich node, edge, and global features. This comprehensive representation is crucial for accurate performance prediction.
- Novel Features Matter: The introduction of specific features like arithmetic intensity and computation/memory access proportions significantly boosts prediction accuracy, highlighting that feature engineering is as important as the model architecture.
- GNNs Excel at Capturing Dependencies: The success of SeerNet demonstrates that GNNs are well-suited for learning the complex interdependencies within DL models that influence performance.
- Aggregation and Boosting Enhance GNNs: SynMM and GNPB are not just minor tweaks; they are shown to significantly improve how GNNs aggregate information and learn from different feature types, leading to better representations.
- Multi-Metric Prediction Needs Careful Handling: Predicting multiple metrics simultaneously is challenging due to conflicting gradients. PCGrad provides an effective solution without sacrificing accuracy or significantly increasing model complexity.
- Low Data Dependency: PerfSeer shows good performance even with a relatively small dataset (1,000 samples), suggesting its robustness and potential for practical deployment.
Key Results
- State-of-the-Art Accuracy: SeerNet achieved an average Mean Absolute Percentage Error (MAPE) of 5.14% on the custom dataset, outperforming baselines like nn-Meter, Brp-NAS, and DIPPM.
- Multi-Metric Performance: SeerNet-Multi achieved a 7.75% MAPE for multi-metric prediction, significantly better than baselines and demonstrating the effectiveness of PCGrad.
- Efficiency: SeerNet uses a minimal number of parameters (1.02M) compared to other GNN-based models, offering a good balance between accuracy and model size.
- Broad Applicability: PerfSeer supports various DL frameworks (via ONNX) and devices (mobile CPUs/GPUs, desktop GPUs, Intel VPUs), showing wide applicability.
- Low Overhead: The representation and prediction latency are low (around 250ms total on an Intel i7 CPU), making it practical for real-time applications.
Limitations and Future Work
- Dataset Specificity: While the new dataset is extensive, its performance metrics were collected on specific hardware (Nvidia RTX 3090). Future work could involve expanding the dataset to include a wider range of hardware and more diverse model architectures.
- Feature Engineering: Although novel features were introduced, there might be further opportunities for feature engineering to capture even more nuanced performance aspects.
- Model Complexity: While SeerNet is efficient, exploring even lighter-weight GNN architectures or prediction models could be beneficial for extremely resource-constrained environments.
- Dynamic Performance: The current work focuses on static performance prediction. Future research could explore predicting performance under dynamic conditions, such as varying batch sizes or network traffic.
Summary updated