TL;DR
This paper introduces a new evaluation framework to test how well Large Language Models (LLMs) perform quantitative data analysis over large collections of unstructured text. By evaluating models on a new benchmark of 470 human-verified questions, the authors reveal a critical architectural bottleneck: when processing more than 500 text instances, open-weights models collapse and even advanced closed-source models struggle significantly with numerical reasoning.
What Problem Does This Paper Address?
While LLMs excel at standard NLP tasks on single text instances, real-world data analysis often requires aggregating insights across thousands of unstructured documents—such as tracking public sentiment trends in a live Twitter feed.
Currently, there is a major gap in how we evaluate these capabilities. Existing data analysis benchmarks focus almost entirely on structured data (like CSV files or SQL databases), while traditional NLP benchmarks focus on single-document question answering. This paper addresses the lack of dedicated datasets and evaluation standards for aggregating information across collections of short, noisy, and independent text instances.
Methods
The authors frame text data analysis as a generative document-based question answering (QA) problem. Instead of a single document, the input is a collection of raw text instances paired with an analytical question.
To evaluate different depths of analytical reasoning, the authors categorize questions into four logical types:
- Existence: Detecting the presence or absence of specific labels (testing for hallucination).
- Count: Maintaining global attention to count specific instances.
- Comparison: Performing binary logical comparisons between label groups.
- Calculation: Executing multi-step arithmetic operations (e.g., deriving percentages).
Models are evaluated under strict formatting constraints (forcing direct answers like "Yes/No", integers, or percentages without explanations). To account for varying data scales and narrow gold-label distributions, the authors implement specialized evaluation metrics: Range-Normalized Root Mean Squared Error (RNRMSE) for percentages and Normalized Root Mean Squared Error (NRMSE) for numerical counts.
Datasets
The authors introduce a new benchmark comprising 470 manually filtered and human-verified question-answer pairs.
The framework is built on top of two well-established social media NLP benchmarks: TweetEval and SuperTweetEval. It integrates eight distinct datasets covering three broad categories of annotation complexity:
- Single-Label Classification (SLC): Sentiment analysis, hate speech detection, and offensive language detection.
- Multi-Label Classification (MLC): Emotion recognition and topic classification.
- Targeted Classification (TC): Stance detection and aspect-based sentiment analysis.
To evaluate scalability, the text sources are randomly sampled into seven distinct size configurations: 10, 50, 100, 250, 500, 750, and 1,000 instances.
Key Insights and Ideas
- The 500-Sample Wall: There is a clear negative correlation between input volume and model performance. Once the input exceeds 500 samples, standard open-weights models experience a sharp performance degradation across all tasks. Even the most capable closed-source models suffer from surging numerical errors at this scale, exposing a universal architectural bottleneck in current LLMs.
- The Reasoning Complexity Gradient: Model performance drops progressively as the task shifts from basic semantic retrieval to mathematical reasoning. Models are highly proficient at Existence questions, but their accuracy drops during Comparisons and plummets on Count and Calculation tasks.
- Semantic vs. Structural Complexity: Tasks with a dual-input structure (like targeted sentiment analysis, which requires conditionally binding a specific entity to a specific sentiment) prove much harder for LLMs than standard single-label classification tasks.
Key Results
- Closed-Source Dominance: Commercial models (Gemini-3.1-Flash-Lite, GPT-5.4-Mini, and Grok-4.1-Fast) consistently lead on numerical regression tasks. For instance, GPT-5.4-Mini achieved the lowest error rates (NRMSE of 0.180) on sentiment datasets.
- Open-Weights Scaling: Smaller open-weights models (<9B parameters, such as LLaMA-3.2-3B and Qwen-3-4B) fail to beat simple heuristic baselines on numerical tasks. However, massive open-weights models like Qwen-3.5-397B perform competitively with closed-source models on binary classification tasks.
- Prompting Strategy Sensitivity: Open-weights models benefit significantly from structured prompting, showing a steady performance increase from "No Instruction" to "Standard" to "One-Shot" prompts. Conversely, One-Shot prompts actually degraded the performance of closed-source models on numerical and percentage tasks.
Limitations and Future Work
- Scale Constraints: Due to the high cost of expert human verification, the benchmark maxes out at 1,000 instances per prompt. It does not address "hyper-scale" analytics involving millions of real-time posts.
- No External Tools: The study evaluates the models' native semantic and numerical reasoning via direct prompting. It does not incorporate agentic workflows or external tools (such as Python code interpreters), which are increasingly common in real-world data science pipelines and would likely establish a performance upper bound.
- Language Bias: The evaluation framework is currently limited to English-language datasets, preventing generalized conclusions for low-resource languages.
Summary updated