TL;DR
This paper tackles the critical problem of detecting "hallucinations" in Large Language Models (LLMs) by proposing an intuitive, human-like approach: ensembling multiple LLM "experts" to judge response consistency. Their method, enhanced by automatic prompt optimization, achieves significant accuracy improvements over baselines, demonstrating the effectiveness of collaborative LLM judgment.
What Problem Does This Paper Address?
Large Language Models (LLMs) are incredibly powerful, but they have a tendency to "hallucinate" – generating text that isn't supported by the input or context. This is a major roadblock for deploying LLMs in applications requiring high accuracy and reliability, such as question answering or summarization. The SemEval-2024 Task 6 specifically focuses on detecting these hallucinations, given the LLM's input task, its generated response, and the ground truth answer. The challenge lies in building robust systems that can reliably distinguish between factual and fabricated information, especially when the LLM's internal workings might be unknown (model-agnostic setting).
Methods
The core of the "Pollice Verso" system is inspired by how humans might verify information: by asking multiple independent "experts" to assess a response. Here, these experts are themselves smaller LLMs.
- Ensemble of Experts: Instead of relying on a single LLM's judgment, the authors use multiple LLMs (specifically, models smaller than 7B parameters) to independently evaluate if a given response is hallucinated.
- Log Probability Analysis: The system doesn't just look at the final "yes" or "no" answer. It analyzes the log probabilities of tokens that indicate a positive ("yes") or negative ("no") judgment. This allows for a more nuanced probability of hallucination, even if "yes" or "no" aren't the most probable single tokens. They use a modified softmax function to combine these probabilities.
- Automatic Prompt Optimization (OPRO): To maximize the effectiveness of each LLM "expert," the authors employ the OPRO framework. This technique uses a "guiding" LLM (ChatGPT 3.5-Turbo) to iteratively refine the prompts given to the expert LLMs. The guiding LLM is trained to generate prompts that lead to higher accuracy on a validation set, effectively "finetuning" the instructions for each expert.
- Ensembling Strategies: The judgments from the individual LLM experts are combined in two ways:
- Uniform Averaging: Simple averaging of their hallucination probabilities.
- Weighted Averaging (Logistic Regression): Training a logistic regression model on the validation set to learn optimal weights for each expert's prediction.
This approach is applied to both the model-aware (where the generating model is known) and model-agnostic (where it's unknown) subtasks, by simply removing model-specific information for the latter.
Datasets
The authors utilized datasets provided by the SemEval-2024 Task 6 organizers. Specifically, they used:
val.model-agnostic.jsonandval.model-aware.v2.jsonfor their internal validation and testing.- They randomly split the
val.model-agnostic.jsoninto "training" and "validation" sets for prompt optimization and logistic regression training, respectively. - The
val.model-aware.v2.jsonwas used as their internal "test" set.
No new benchmarks were introduced; they focused on improving performance on the existing task.
Key Insights and Ideas
- Human-like Collaboration: The core idea of using multiple LLM "experts" to collectively judge a response is a powerful analogy to human verification processes and proves effective.
- Beyond "Yes/No": Relying on the probability distribution of tokens, rather than just the single most probable token, provides a more robust and reproducible measure of hallucination likelihood.
- Prompt Engineering Automation: OPRO significantly boosts performance by automatically finding optimal prompts, reducing the manual effort and guesswork typically involved in prompt engineering.
- Small Models Can Shine: Surprisingly, the smaller Zephyr 3B model performed exceptionally well, even outperforming larger models in some configurations. This suggests that model size isn't the sole determinant of performance for this specific task.
- Model-Agnostic Advantage: The authors argue that the model-agnostic setting, while potentially harder, offers better transferability in practice, as it doesn't rely on specific model architectures.
Key Results
The "Pollice Verso" system achieved notable improvements:
- Accuracy Gains: Up to 10.6 percentage points (p.p.) improvement compared to the challenge baseline.
- Model-Agnostic Performance: Their best ensemble achieved an accuracy of 80.3% on the model-agnostic task, ranking 41st out of 260 submissions.
- Model-Aware Performance: For the model-aware task, their best ensemble achieved 0.777 accuracy, ranking 103rd out of 295 submissions.
- Zephyr 3B Dominance: The Zephyr 3B model, often in conjunction with Zephyr 7B, was a consistent component of the best-performing ensembles.
Limitations and Future Work
- Runtime: While the system is flexible, runtime can be a concern depending on the number of models in the ensemble. Future work could explore runtime optimizations like batching prompts or investigating different model quantization methods.
- Ground Truth Dependency: The current system relies on the ground truth answer. The authors suggest that future work could adapt the prompt to remove the ground truth, allowing the system to function as a validator that checks if a model output is a correct answer to the input task, rather than just being grounded in a provided truth.
- Joint Prompt Optimization: Currently, prompts are optimized independently for each model. Jointly optimizing prompts for an entire ensemble could lead to further improvements.
- Exploring Other Quantizations: Investigating different quantization methods for LLMs could offer a trade-off between model size, speed, and performance.
Summary updated