Abstract
Artificial Intelligence models frequently fail to grasp fundamental "commonsense" facts because human communication is subject to reporting bias—the tendency to only document novel or noteworthy information. This study explores whether children’s literature, which explicitly explains the world to new learners, can mitigate this bias. By evaluating the density of commonsense assertions across various corpora, the researchers found that texts with higher readability scores contain significantly more basic facts. Fine-tuning a BERT model on a small, 2 MB corpus of infant-targeted books (InfantBooks) yielded superior performance in commonsense probing and typicality ranking compared to models trained on massive, general-purpose web datasets like C4.
Introduction: The Challenge of Reporting Bias
In Natural Language Processing (NLP), the prevailing wisdom is that larger datasets lead to better models. However, Commonsense Knowledge (CSK) acquisition is hindered by reporting bias. Humans rarely write down the obvious, such as "the sun is bright" or "water is wet." Instead, we focus on exceptions or complex metaphors. Consequently, models trained on standard web crawls or Wikipedia often learn a skewed reality where rare events are statistically overrepresented relative to fundamental truths.
The authors hypothesize that we can bypass this bias by turning to texts written for children. These books are specifically designed for individuals who lack a baseline understanding of the world, making them a potential goldmine for explicit commonsense assertions.
Methodology
The researchers employed a two-pronged strategy involving density analysis and model evaluation:
- Density Analysis: Using the Flesch Reading-ease score (FRE)—a metric based on sentence and syllable length—they categorized the simplicity of various texts and mapped them against known commonsense assertions.
- Model Fine-tuning: A standard BERT-large model was fine-tuned on different corpora to compare the impact of data source on knowledge acquisition.
- Evaluation Probes: The models were tested using LAMA (fill-in-the-blank probes) and COMET (generative subject-relation pairs) across benchmarks like ConceptNet and Quasimodo.
Dataset Characteristics
The study compared four primary sources: the massive C4 web crawl, a high-readability subset called C4-easy, the Children Book Test (CBT), and a new, highly specialized InfantBooks dataset.
As shown in Table 1, the InfantBooks corpus, while small in total volume, possesses a significantly higher readability score (91) compared to the standard C4 web text (60).
The Commonsense Density Advantage
The core finding of the density analysis is a direct correlation between text simplicity and the frequency of commonsense facts. The InfantBooks corpus proved to be exceptionally dense in this regard.
As illustrated in Table 2, the InfantBooks corpus contains over five times more commonsense assertions per word than the general C4 sample. This suggests that "simplicity" in text serves as an effective proxy for the explicit statement of fundamental world knowledge.
Key Results: Quality Over Quantity
The experimental results challenge the "bigger is better" paradigm. Fine-tuning on just 2 MB of InfantBooks data outperformed models trained on gigabytes of general web data in several key areas:
- Probing Accuracy: On the CSLB property norms dataset, the InfantBooks-tuned model achieved a Mean Reciprocal Rank (MRR) of 0.174, surpassing both the vanilla BERT (0.149) and the C4-tuned model (0.137).
- Typicality Ranking: Children's books excel at teaching "typical" assertions (e.g., "lions drink water") over "salient but atypical" ones (e.g., "lions attack humans").
- Human Preference: In blind evaluations, humans consistently preferred the predictions of the InfantBooks model. For instance, when asked what "Bears" do, the model suggested "bite, eat, walk," whereas the standard model produced sub-word artifacts like "##kin."
Limitations and Future Work
Despite its effectiveness, the InfantBooks approach faces scaling challenges. The current corpus is small (496 books), and copyright restrictions limit the creation of massive open-source versions. Furthermore, while the Flesch score is a useful metric, it is an imperfect proxy for educational intent. Future research may focus on identifying "explanatory" content within larger web crawls to combine the benefits of scale with the density of children's literature.
Conclusion
The "key" to artificial commonsense may not lie in larger models or more massive web scrapes, but in the simple, foundational stories we use to teach our children. By prioritizing the quality and explanatory nature of training data, we can build AI that better understands the "obvious" world around us.
Summary updated