Introduction
Large Language Model (LLM) performance is notoriously sensitive to both the specific model architecture and the phrasing of the prompt. In the current era of "LLM abundance," users have access to dozens of models, yet no single model dominates every task. A model might excel at logical reasoning but struggle with creative rephrasing, or it might be "brittle"—performing well on one prompt template while failing if a single word in the instructions is changed.
The ideal solution is to select the optimal model and prompt for each individual query. While "semantic confidence" (the consistency of a model's answers across multiple samples) is a reliable proxy for correctness, calculating it at runtime for every possible model-prompt combination is computationally prohibitive. This paper introduces PICS (Predicted-Instance-Confidence Search), a framework that automates this selection process using lightweight auxiliary models to predict performance without the need for exhaustive inference.
The PICS Framework
PICS shifts the heavy computational burden from inference time to an offline precomputation phase. The core innovation lies in using a neighbor-based regression approach to estimate how confident a model will be on a specific, unseen query.
The Pipeline Steps:
- Offline Confidence Mapping: A training dataset is used to precompute the actual semantic confidence for every available model-prompt pair.
- Neighbor-Based Featurization: For a new query, the system finds its $k$-nearest neighbors in the training set using text embeddings. This provides a stronger signal than raw text alone by looking at how models performed on similar past queries.
- Confidence Regression: An ensemble of small, fast regressors (e.g., Random Forest, Gradient Boosting) predicts the confidence for all available pairs based on the neighbors' performance data.
- Selection and Refinement: The pair with the highest predicted confidence is chosen. In the PICS-CS (Confidence Sampling) variant, the selected model generates multiple responses, and a final semantic consistency check is performed to return the most reliable answer.
Experimental Evaluation
The authors evaluated PICS across three diverse tasks:
- StrategyQA: Multi-step logical reasoning.
- TriviaQA: Factual, open-domain question answering.
- Question Formation: Linguistic mapping (e.g., converting statements to questions).
These tasks were tested using four models of varying scales: FLAN-T5-XL (11B), FLAN-UL2 (20B), MPT-30B, and BLOOM (176B).
Key Results
PICS-CS consistently outperformed the "Most Confident" baseline—the single model-prompt pair that performs best on average for a given dataset. The results demonstrate that PICS effectively routes queries to the "subject matter expert" model and prompt.
Major Insights:
- Prompt Sensitivity: The impact of prompt selection is profound. For the MPT-30B model on the Question Formation task, PICS-CS improved accuracy from a baseline of 42% to nearly 80% by selecting the optimal template for each query.
- Prediction as Regularization: Interestingly, predicted confidence sometimes outperformed the "Oracle" (actual calculated confidence). The regression models appear to act as smoothers, ignoring "lucky" high-confidence guesses on outliers and focusing on generalizable patterns.
- Efficiency: Runtime analysis indicates that PICS is nearly as fast as random model selection, whereas an exhaustive Oracle search is orders of magnitude slower and impractical for real-time applications.
Conclusion and Future Work
PICS provides a scalable, high-accuracy method for navigating the complex landscape of LLMs and prompts. While highly effective, the system currently relies on the query being somewhat similar to the training data. Future research will focus on scaling the search space to even larger sets of models and exploring more advanced embedding techniques to further refine prediction accuracy for outlier queries.
Summary updated