TL;DR
This paper introduces "multi-problem evaluation" to probe Large Language Models (LLMs) by presenting them with multiple classification tasks within a single prompt. They discover that while LLMs excel at batch classification, they significantly falter on index selection tasks that are conceptually similar, suggesting a limitation in their ability to combine classification with precise index identification.
What Problem Does This Paper Address?
Large Language Models (LLMs) have shown impressive performance on various benchmarks, raising concerns about potential data contamination during training. To address this, researchers are exploring ways to evaluate LLMs on tasks that are less likely to have appeared in their training data. This paper focuses on a specific aspect of LLM capabilities: their ability to handle multiple, related problems simultaneously. While LLMs are often evaluated on single tasks, this work investigates how their performance degrades when presented with multiple, homogeneous tasks bundled into a single prompt, particularly when these tasks require not just classification but also the precise identification of specific elements within the text.
Methods
The core innovation of this paper is the multi-problem evaluation approach. Instead of prompting an LLM with one problem at a time, they construct prompts containing multiple, homogeneous classification problems. They then categorize these into four evaluation task types:
- Single Classification (SingleClf): The standard approach, one problem per prompt.
- Batch Classification (BatchClf): Multiple classification problems presented in a single prompt, with each problem expecting a separate answer.
- Index Selection One Category (SelectOne): LLMs are asked to identify the indices of text that belong to a specific category, presented for each category separately.
- Index Selection All Categories (SelectAll): LLMs are asked to identify the indices of text belonging to all categories within a single prompt.
They leverage existing classification benchmarks and adapt them to these multi-problem formats.
Datasets
The study utilizes six existing classification benchmarks:
- SST-2 (Sentiment Analysis)
- COLA (Grammatical Acceptability)
- AGNews (Topic Classification)
- MRPC (Paraphrase Detection)
- SNLI (Natural Language Inference)
- WiC (Word Sense Disambiguation)
For AGNews and SNLI, they sampled 1,000 examples from the test splits. They also created a novel synthetic dataset based on SST-2 to further mitigate data contamination concerns.
Key Insights and Ideas
- LLMs are good at batch processing, but struggle with precision: LLMs can handle multiple classification tasks simultaneously (Batch Classification) almost as well as they do individually. This suggests a strong capability for parallel processing of similar tasks.
- Index selection is a bottleneck: The significant performance drop observed in the "Index Selection" tasks (SelectOne and SelectAll) points to a core limitation. LLMs struggle to accurately combine the act of classification with the precise identification and outputting of specific text indices.
- Conceptual equivalence doesn't guarantee performance: The SelectOne and SelectAll tasks are conceptually very similar to Batch Classification, yet LLMs perform much worse on them. This suggests that the format of the output and the specific instruction to pinpoint indices is the challenge, rather than the underlying classification itself.
- This limitation is widespread: The performance drop is observed across all tested LLMs, regardless of whether they are used in zero-shot, few-shot, or Chain-of-Thought (CoT) settings. This suggests a fundamental capability limitation rather than a prompt engineering issue.
- Multi-problem evaluation is a powerful diagnostic tool: This evaluation paradigm effectively uncovers limitations that might be masked by standard single-problem evaluations.
Key Results
- LLMs maintain over 90% of their SingleClf accuracy on average for BatchClf tasks. The overall accuracy drop from SingleClf to BatchClf is a minor 3.2%.
- However, for the index selection tasks, the performance drop is substantial. SelectOne shows a 32.0% accuracy difference compared to BatchClf, and SelectAll shows a 12.1% difference. These differences are statistically significant with large effect sizes.
- The performance gap between SelectOne and SelectAll is also significant, with SelectAll generally performing better.
- Even with simplified index selection tasks (where text lines are directly replaced by their labels), LLMs perform much better, indicating that the difficulty lies in the combination of classification and index selection, not just index selection itself.
- Chain-of-Thought (CoT) prompting offers some improvement for index selection tasks, but does not fundamentally alter the task complexity hierarchy. Few-shot prompting, surprisingly, can be detrimental.
Limitations and Future Work
The authors acknowledge that their study is limited by the cost, time, and space constraints, leading to a selection of experiments rather than an exhaustive one. They also note that they inherit any labeling errors from the pre-existing benchmarking datasets. Future work could explore:
- Testing LLMs on more diverse problem types (e.g., reasoning) and heterogeneous problems (mixing different tasks/benchmarks).
- Investigating the causes of performance variations across different benchmarks, task sizes, and LLM architectures.
- Conducting model-level ablation studies to understand how LLMs acquire multi-problem handling abilities and how to improve them.
Summary updated