TL;DR
This paper introduces "GuessingGame," a novel protocol for evaluating how well Large Language Models (LLMs) can ask strategic, open-ended questions to identify a hidden object. They propose two information gain metrics that effectively predict question quality and show that targeted prompting can significantly improve LLM performance, demonstrating that question-asking ability is measurable and improvable.
What Problem Does This Paper Address?
While LLMs have shown remarkable capabilities in understanding and generating text, their ability to ask effective questions, especially in interactive, open-domain settings, remains underexplored. Existing evaluation methods often rely on constrained question formats (like yes/no) or predefined answer spaces, which don't reflect real-world scenarios. This limitation hinders the development of LLMs for applications requiring adaptive information gathering, such as education, medical diagnosis, or autonomous decision-making. The paper aims to bridge this gap by providing a robust framework to measure and improve the strategic question-asking abilities of LLMs.
Methods
The core of the paper is the GuessingGame protocol. It's a multi-agent setup involving:
- Oracle: An LLM that knows a secret object.
- Guesser: An LLM that asks free-form questions to identify the object.
- Checker: An LLM that validates question types and enforces experiment constraints.
To quantify question quality, two Information Gain (IG) metrics are proposed:
- Bayesian Belief Update: This metric uses an "Interpreter" LLM to score the relevance of an answer to various semantic concepts. These scores are then used to update a belief distribution over concepts using a Bayesian-style rule. The information gain is measured by the KL divergence between the belief distribution before and after receiving an answer, reflecting how much the answer shifted the model's understanding.
- Entropy-Based Information Gain: This method leverages the ConceptNet knowledge graph. Oracle answers are converted into embeddings, and their similarity to ConceptNet concepts is used to identify implied assertions about the object. These assertions are then used to prune a set of candidate objects. Information gain is calculated as the reduction in entropy (logarithm of the number of remaining candidates) before and after the pruning.
The paper also defines a five-part taxonomy of question types: Attribute, Function, Location, Category, and Direct guesses.
Datasets
The secret objects used in the GuessingGame experiments were drawn from Jiang and Riloff (2021), a collection of everyday objects annotated with typical functions. This corpus was filtered to ensure standalone objects, resulting in 858 distinct objects used across all experiments.
Key Insights and Ideas
- Open-ended questions are king: The paper strongly advocates for open-ended questions, showing they yield significantly higher information gain and better performance than binary (yes/no) questions. This is because they elicit richer responses that provide more nuanced information.
- Information Gain predicts efficiency: Higher information gain directly correlates with faster game completion (fewer questions needed), demonstrating its utility as a measure of question quality.
- Prompting matters immensely: Even weaker LLMs can achieve impressive performance if prompted correctly. Simple interventions like preventing repetitive question types or forcing open-ended questions lead to dramatic improvements.
- Attribute questions are highly informative: Questions about an object's physical attributes (shape, size, color) consistently yield the highest information gain, suggesting they are the most effective for narrowing down possibilities early in the game.
- Bayesian IG is more descriptive: While both IG metrics are useful, the Bayesian approach offers a more flexible and descriptive measure of question utility, correlating better with task efficiency.
Key Results
- LLaMA-3.3 70B performance: In the standard open-ended setting, LLaMA-3.3 70B achieved a 39.4% success rate. This jumped to 80.0% when a "repeat-type constraint" was applied and to 97.4% when "forced open-endedness" was enforced.
- Information Gain and Game Length: A one-standard-deviation increase in Bayesian IG was associated with a 43% reduction in expected game length. The entropy-based IG showed a 19% reduction.
- Question Type Performance: Attribute-only questions achieved a 35.8% success rate, nearly matching the full-question baseline, and yielded the highest average IG.
- Proprietary Models: GPT-40 and Gemini 2.0 Flash-Lite outperformed LLaMA-3.3 70B out-of-the-box (64.1% and 74.1% SR, respectively), but all models showed significant improvement with forced open-ended prompting.
- Human Performance: Human participants also showed a strong correlation between information gain and game efficiency, with experts achieving a 96.3% success rate in an average of 7.00 questions.
Limitations and Future Work
- External vs. Internal Belief Modeling: The Bayesian IG metric relies on an external "Interpreter" LLM to score concept relevance, rather than directly accessing the Guesser LLM's internal beliefs. This means it measures observable behavior, not necessarily latent cognition. Future work could investigate how these external belief updates align with internal representations.
- Interpreter LLM Dependence: The Bayesian IG metric's accuracy is dependent on the Interpreter LLM's calibration. Biases or errors in the Interpreter could lead to misleading belief updates.
- Knowledge Base Coverage: The entropy-based IG metric relies on ConceptNet, which has limitations in coverage for niche objects and may contain sparse or noisy data.
- Domain Scope: Experiments were limited to everyday, prototypical objects. The framework could be extended to more abstract or high-stakes domains like medical diagnosis or scientific discovery.
- Generalizability of IG Metrics: While correlations with human performance are promising, further work is needed to establish if the IG metrics are truly domain-general or human-aligned measures of informativeness.
Summary updated