TL;DR
Researchers often struggle to efficiently compare different Large Language Models (LLMs) due to the complexities of interacting with various APIs and managing rate limits. This paper introduces "Prompto," an open-source Python library that simplifies this process by enabling asynchronous, parallel querying of multiple LLM endpoints, significantly speeding up experimentation and reproducibility.
What Problem Does This Paper Address?
The rapid proliferation of Large Language Models (LLMs), both through commercial APIs (like OpenAI and Gemini) and self-hosted "open" models (like Llama and Gemma), presents a fantastic opportunity for research. However, this also creates a significant hurdle: efficiently interacting with these diverse models. Each LLM endpoint often requires custom code for communication, and managing their individual rate limits (queries per minute, or QPM) adds another layer of complexity. This makes comparative studies, which are crucial for understanding LLM capabilities and limitations, incredibly time-consuming and hinders reproducibility. Researchers need a streamlined way to send queries to multiple LLMs concurrently, manage their individual constraints, and collect results systematically.
Methods
Prompto tackles this challenge by leveraging Python's asyncio library for asynchronous programming. This allows the library to send multiple requests to different LLM endpoints concurrently without waiting for each individual response. Key features include:
- Unified Interface: Prompto provides a consistent way to interact with various LLM APIs (e.g., OpenAI, Gemini, Anthropic) and self-hosted models (e.g., via Ollama).
- Asynchronous Querying: Enables sending multiple requests simultaneously, drastically reducing idle wait times and maximizing efficiency, especially when dealing with different rate limits.
- Parallel Processing: Supports querying multiple APIs or models in parallel within a single experiment run, further accelerating data generation.
- Experiment Management: Prompts, model names, and parameters are defined in a single JSON Lines (JSONL) file, promoting reproducibility by keeping all experiment inputs in one place.
- Automatic Evaluation: Includes built-in functionalities for evaluating model responses, such as string matching, regex matching, and support for LLM-as-a-judge evaluations.
- Extensibility: Designed to be easily extended to integrate new LLM APIs and models by creating custom classes.
Datasets
The paper does not introduce new datasets for training. Instead, it uses a sample of 100 prompts from the instruction-following dataset provided by Taori et al. (2023) for their experiments. This dataset is used as input to various LLMs to generate responses for evaluation.
Key Insights and Ideas
- Asynchronous is Key for LLM Efficiency: The core insight is that traditional synchronous request-response patterns are highly inefficient for LLMs, especially when dealing with multiple models or strict rate limits. Asynchronous programming is essential for maximizing throughput.
- Reproducibility Through Standardization: Defining all experiment parameters (prompts, models, settings) in a single, standardized file format (JSONL) is a simple yet powerful way to ensure that experiments can be easily replicated by others.
- Parallelism Dictated by the Slowest: When querying multiple LLMs in parallel, the overall run-time is largely determined by the slowest model or API. This highlights the importance of efficient individual endpoint handling even within a parallel framework.
- Beyond Evaluation: While Prompto is excellent for evaluation, its ability to generate synthetic data and facilitate model distillation opens doors for broader LLM development workflows.
Key Results
The paper demonstrates significant speedups in querying LLM endpoints:
- Single API, Multiple Models: When querying OpenAI's GPT-3.5-turbo, GPT-4, and GPT-4o models, Prompto achieved an approximate 35x speedup compared to a synchronous approach for obtaining 100 responses.
- Multiple APIs: For querying OpenAI, Gemini, and Ollama, Prompto achieved a 2x speedup when running queries in parallel for a total of 300 prompts.
- Individual API Speedups: For OpenAI and Gemini, Prompto showed speedups of around 9x and 11.6x respectively for 100 responses. For Ollama, the speedup was modest (1.01x) due to Ollama's internal queuing mechanism.
Limitations and Future Work
The authors acknowledge several areas for future development:
- Batch Inference for Local Models: While Prompto handles API rate limits well, there's ongoing work to improve efficiency for local models, potentially by integrating with batch inference strategies or toolkits like vLLM.
- Advanced Evaluation: Expanding evaluation capabilities to include more computationally intensive scoring functions (e.g., toxicity classifiers) and integrating with data visualization libraries for better performance analysis.
- Translation Pipelines: Enhancing rephrasing pipelines for tasks like prompt translation, which could be valuable for synthetic data generation and improving multilingual machine translation.
- Direct Model Integration: The ultimate goal is to enable direct model instantiation within Prompto, bypassing the need to set up API endpoints for local models, further streamlining experimentation.