TL;DR
This paper explores the hidden biases of ChatGPT, revealing that when the model "hallucinates" digits in complex math problems, it doesn't do so randomly. Instead, its digit frequency closely mirrors human psychological preferences—specifically a bias toward the number 7—rather than following mathematical distributions like Benford’s Law.
What Problem Does This Paper Address?
While Large Language Models (LLMs) like ChatGPT are celebrated for their conversational prowess, they are notoriously poor at "system 2" tasks like precise mathematical calculation. When asked to compute complex values (like roots or powers of irrational numbers), ChatGPT often provides a confident but entirely incorrect string of digits.
The author, Amos Azaria, investigates a fascinating question: when ChatGPT makes up these numbers, where do the digits come from? By analyzing these errors, the paper seeks to understand the underlying statistical biases inherited from the human-generated data the model was trained on. Additionally, the paper explores how ChatGPT’s conversational nature serves as both a strength (allowing for error correction) and a weakness (leading to overconfidence and framing sensitivity).
Methods
The researcher employed a clever "stress test" to extract raw digit distributions from ChatGPT. He posed mathematical queries that result in irrational numbers (e.g., $pi \times 4.26$ or the third root of 2811) and requested the model provide between 100 and 400 digits of precision.
Because these specific calculations were unlikely to be in the training set, ChatGPT was forced to "sample" or generate digits. To ensure the data reflected the model's internal bias rather than actual mathematical truth, the author:
- Removed the first five digits of every response (which might be approximately correct).
- Filtered out "lazy" sequences (like long strings of the same digit or "123456789").
- Analyzed the frequency of the remaining 10,811 digits.
The resulting distribution was then compared against two benchmarks: Benford’s Law (which describes the frequency of digits in natural datasets) and a human survey of "favorite numbers."
Datasets
The paper does not introduce a new machine learning training dataset. Instead, it generates a novel evaluation dataset consisting of 10,811 digits produced by ChatGPT in response to complex math prompts. For comparison, the author uses:
- Bellos (2015): A survey of human favorite numbers (1–9) to see if ChatGPT’s "random" choices align with human psychology.
- Standard Game Theory Benchmarks: The Ultimatum Game, Prisoner’s Dilemma, and Trust Game were used to evaluate ChatGPT’s social and "rational" decision-making.
Key Insights and Ideas
- The "Lucky 7" Bias: In a perfectly random distribution of an irrational number, each digit should appear ~10% of the time. ChatGPT, however, showed a massive spike for the digit 7 (13.28%) and a significant deficit for the digit 1 (8.73%).
- Mirroring Human Psychology: The correlation between ChatGPT’s digit frequency and human "favorite numbers" is a staggering 0.893 (Pearson coefficient). This suggests that ChatGPT isn't just a statistical model of language; it is a statistical model of human preference.
- Defiant Overconfidence: When challenged on a wrong math answer, ChatGPT doesn't just fail; it often doubles down. In one example, it incorrectly calculated a large multiplication and, when corrected, essentially told the user they didn't understand how exponents worked to justify its own hallucination.
- Framing is Everything: In the "Ultimatum Game," ChatGPT’s behavior changed based on a single word. If told a player offered "$100$," it accepted. If told the player offered "only $100$," it rejected the offer, viewing it as "unfair."
Key Results
- Digit Frequency: The digit 7 was the most frequent, matching the most popular "favorite number" in human surveys. The digit 1 was the least frequent, matching the least popular human favorite.
- Benford’s Law Violation: Unlike natural data (where '1' is the most common leading digit), ChatGPT’s output showed no adherence to natural numerical laws, confirming its bias is purely "human-centric."
- Social Games: In the Prisoner’s Dilemma and Trust Game, ChatGPT consistently chose "cooperative" or "fair" actions rather than the "rationally selfish" actions predicted by classical economic theory. It behaved like a typical human participant rather than a cold, calculating agent.
Limitations and Future Work
The author notes several limitations that provide a roadmap for future research:
- In-Context Learning: Despite being a "conversational" agent, the version of ChatGPT tested failed to learn simple instructions within a single session (e.g., failing to follow a command to say a specific phrase when greeted).
- Version Specificity: The study was conducted on an early version of ChatGPT. As models are updated with RLHF (Reinforcement Learning from Human Feedback), these biases might shift or be "patched," though the underlying training data remains human-centric.
- Sample Size: While 10,000+ digits is significant, larger-scale studies across different LLMs (like Claude or Gemini) would be needed to see if this "human favorite" bias is a universal property of transformer-based models.
The paper concludes that while ChatGPT is an incredible tool for dialogue, its "mathematical" output is often just a reflection of human whimsy, making it a better mirror of our psychology than a reliable calculator.
Summary updated