Introduction
While Large Language Models (LLMs) have demonstrated an impressive ability to recall specific facts, their understanding of the relative popularity or "social visibility" of cultural concepts remains under-explored. Most evaluations focus on micro-level knowledge—such as naming a capital city—rather than macro-level knowledge, which involves understanding the statistical distribution of concepts within a culture. For social scientists and digital humanists, determining whether an LLM captures these corpus-level trends is essential for assessing how models represent different societies.
This paper introduces CPopQA, a benchmark designed to evaluate whether LLMs can rank cultural concepts—specifically holidays—by their statistical prevalence. The study reveals that while advanced models show some intuitive grasp of cultural proximity, they struggle with "long-tail" concepts and often fail to outperform simple heuristics like Wikipedia article length.
The CPopQA Benchmark
The authors propose the Cultural Popularity Question Answering (CPopQA) task. In this challenge, a model is given a set of $n$ holidays (where $n \in \{2, 3, 5\}$) and must rank them in descending order of popularity within a target country, such as the US or the UK.
To establish a robust ground truth for "popularity," the researchers utilized the Google Books Ngram Viewer (GBNV). They calculated the average frequency of holiday names across American and British English book corpora spanning 220 years (1800–2019). The resulting dataset includes 457 holidays from 58 countries, ensuring geographic diversity by including both high-population and low-population nations across five continents. In total, 9,000 QA testing pairs were generated through various holiday combinations.
Performance Analysis
The study evaluated four primary models: GPT-3.5 (text-davinci-003), LLaMA-7B, LLaMA-13B, and BLOOM-7b1. These were compared against three baselines: random guessing, Google Trends data ("google stat"), and Wikipedia article length ("wiki len").
As shown in the results, GPT-3.5 was the top-performing model, particularly as the complexity of the ranking task increased. However, a significant "complexity penalty" was observed: as the number of items to rank grew, exact match accuracy plummeted. For 5-item ranking, GPT-3.5 achieved only 6.6% accuracy, although its Precision@1 remained relatively high at 62%. Notably, the "wiki len" baseline proved remarkably formidable, often outperforming the LLMs in 2-item ranking. This suggests that the sheer volume of documentation for a concept is a powerful proxy for popularity that LLMs have not yet fully internalized as a statistical signal.
Cultural and Statistical Biases
A key finding of the research is the persistent struggle with "long-tail" cultural concepts. Models performed significantly better when ranking holidays from high-population countries compared to those from low-population ones. This indicates a systemic bias toward well-represented cultures in the training data.
Furthermore, the data suggests a phenomenon of "geo-cultural proximity." GPT-3.5, for instance, was more accurate at ranking holidays from regions culturally similar to the US (such as Oceania) than from more distant cultures (such as Asia). This implies that LLMs may implicitly learn a "cultural map" based on the frequency and context of mentions in their training corpora. Interestingly, scaling parameters did not always yield better results; LLaMA-13B occasionally underperformed the smaller LLaMA-7B, suggesting that a model's grasp of nuanced cultural statistics is not solely a function of size.
Conclusion
CPopQA highlights a critical gap in LLM capabilities: the transition from knowing a fact to understanding its relative cultural weight. While models like GPT-3.5 show promise, the dominance of simple baselines like Wikipedia article length suggests that LLMs are not yet optimal at reasoning about the statistical "vibe" of human culture. Future work will focus on expanding popularity metrics to include more diverse web-based resources and developing prompting strategies to improve statistical reasoning regarding under-represented cultures.
Summary updated