LM4HPC: Towards Effective Language Model Application in High-Performance Computing

Le Chen, Pei-Hung Lin, Tristan Vanderbruggen, Chunhua Liao, Murali Emani, Bronis R. de Supinski

TL;DR

This paper introduces LM4HPC, a novel framework designed to bridge the gap between powerful language models (LMs) and the specialized needs of High-Performance Computing (HPC). LM4HPC provides HPC-specific tools, datasets, and pipelines, enabling researchers to more easily leverage LMs for tasks like code analysis and optimization, and demonstrating competitive performance of general LMs on these tasks.

What Problem Does This Paper Address?

The rapid advancement of large language models (LLMs) like GPT-4 has opened up exciting possibilities across many fields. However, applying these LLMs to the domain of High-Performance Computing (HPC) software analysis and optimization presents significant challenges. Existing LLM frameworks are primarily designed for natural language processing and lack the specific support needed for HPC tasks. This includes a scarcity of HPC-specific datasets, the difficulty of pre-processing code for LLMs, limitations in handling large code inputs, and a lack of standardized evaluation processes for comparing different models on HPC-related problems. Consequently, it's hard for HPC researchers and developers to effectively utilize LLMs for their work.

Methods

The core contribution of this paper is the LM4HPC framework. This framework acts as an intermediary, making LLMs more accessible and useful for HPC tasks. It's built upon existing machine learning components, notably with Hugging Face-compatible APIs, to ensure a familiar user experience. LM4HPC offers:

  • Extensible Components: It allows for the inclusion of relevant machine learning components tailored for HPC.
  • HPC-Specific APIs: These APIs facilitate essential operations like code preprocessing, tokenization, integration of new data, and evaluation.
  • Specialized Pipelines: The framework includes pre-built pipelines for common HPC tasks such as code similarity analysis, parallelism detection, and OpenMP question answering.
  • Standardized Evaluation: LM4HPC introduces standardized workflows and metrics to enable fair and reproducible evaluations of LLMs for HPC tasks, aiming to create insightful leaderboards.
  • Integration with LangChain: To address the challenge of incorporating new data, LM4HPC leverages the LangChain framework, enabling LLMs to access and utilize external knowledge bases like OpenMP documentation.

Datasets

LM4HPC utilizes and contributes to HPC-specific datasets. For evaluation, the paper uses:

  • POJ-104: A dataset derived from a pedagogical programming system, useful for code similarity tasks.
  • DRB-ML: A dataset of C/C++ OpenMP kernels, extended with labels indicating parallelizability.
  • OMP4Par: An open-source benchmark containing OpenMP code with loop parallelism labels.
  • OMPQA: A newly created dataset specifically for OpenMP question answering, designed to probe LLM capabilities in this domain.

Key Insights and Ideas

  • General LLMs are surprisingly capable: Even models not specifically trained for code (like GPT-3.5-turbo) show strong performance on HPC tasks, often outperforming specialized code models. This suggests a broad applicability of LLMs beyond their initial design.
  • HPC needs its own tools: The lack of HPC-specific datasets, tokenizers, and evaluation metrics is a significant barrier. LM4HPC directly addresses this by providing a tailored ecosystem.
  • Code tokenization is tricky: Standard NLP tokenizers can misinterpret code constructs (e.g., splitting variable names), highlighting the need for code-aware tokenization.
  • Input size is a major bottleneck: LLMs' limited input token lengths pose a challenge for analyzing large HPC codebases, requiring careful snippet extraction and context management.
  • Standardized evaluation is crucial: The paper emphasizes the need for reproducible benchmarks and leaderboards to guide model selection and research in this emerging area.

Key Results

The paper evaluates LM4HPC on three representative tasks:

  • Code Similarity Analysis: GPT-3.5-turbo demonstrated superior performance, outperforming traditional code models like CodeBERT and GraphCodeBERT. StarChat-Alpha and Dolly 2.0 also showed commendable results.
  • Parallelism Detection: GPT-3.5-turbo again led, achieving an F1 score of 93.9 on the OMP4Par dataset, indicating LMs' effectiveness in identifying parallelism opportunities. However, specialized models like augAST showed better precision.
  • OpenMP Q&A: GPT-3.5-turbo generally performed best, though all models showed room for improvement, with ROUGE-L F1 scores around 0.259. Augmenting models with OpenMP documentation via LangChain showed slight improvements for certain question categories.

Limitations and Future Work

The authors acknowledge several limitations and suggest future directions:

  • Input Size Limitation: Handling large HPC codebases remains a challenge due to token limits.
  • Dataset Availability: The need for more HPC-specific training and evaluation datasets is highlighted.
  • Evaluation Metrics: Current metrics for code evaluation are not as universally accepted as those for natural language.
  • Future Work: The team plans to explore automated dataset generation for HPC, enhance LM4HPC for fine-tuning models on HPC tasks, and provide performance analysis and optimization suggestions. They also aim to incorporate GPT-4 once API access is widely available.

Summary updated

Abstract

In recent years, language models (LMs), such as GPT-4, have been widely used in multiple domains, including natural language processing, visualization, and so on. However, applying them for analyzing and optimizing high-performance computing (HPC) software is still challenging due to the lack of HPC-specific support. In this paper, we design the LM4HPC framework to facilitate the research and development of HPC software analyses and optimizations using LMs. Tailored for supporting HPC datasets, AI models, and pipelines, our framework is built on top of a range of components from different levels of the machine learning software stack, with Hugging Face-compatible APIs. Using three representative tasks, we evaluated the prototype of our framework. The results show that LM4HPC can help users quickly evaluate a set of state-of-the-art models and generate insightful leaderboards.
BibTeX
@inproceedings{chen2023lm,
  author = {Le Chen and Pei-Hung Lin and Tristan Vanderbruggen and Chunhua Liao and Murali Emani and Bronis R. de Supinski},
  title = {{LM4HPC: Towards Effective Language Model Application in High-Performance Computing}},
  booktitle = {IWOMP},
  year = {2023},
  url = {https://openreview.net/forum?id=51uQ6NZdQd},
}
Figure