Abstract
Large Language Models (LLMs) serve as vast repositories of factual information, yet the process by which they acquire this knowledge during pretraining remains largely opaque. This study investigates the developmental trajectory of factual recall in OLMo-7B, specifically focusing on how and when facts are learned across different languages and the role of cross-lingual transfer. By tracing the model across 400,000 training steps, we find that factual acquisition is a rapid, frequency-driven process that leverages cross-lingual representational alignment to bridge data gaps.
Introduction
While LLMs are often treated as finished products, their training is a dynamic process. A significant challenge in understanding these models is the "cross-lingual knowledge barrier": why a model might correctly answer a query in English but fail in another language. This research addresses whether this failure stems from a lack of exposure in specific languages or a failure to transfer knowledge across the language barrier.
Methodology
The researchers conducted a longitudinal study of OLMo-7B, an open-source LLM, analyzing its state at checkpoints every 1,000 steps during early training. The study utilized several key tools and datasets:
- KLAR Dataset: A multilingual factual probing dataset containing 1,197 facts across 12 relation categories (e.g.,
capital_of,manufacturer) in 12 diverse languages. - Dolma Corpus: The 3-trillion-token English-centric corpus used for OLMo's training.
- WIMBD (What's In My Big Data?): Used to count subject-object co-occurrences in the training data to determine frequency.
- Similarity Dynamics: The team measured cosine similarity between internal representations of queries in different languages to track how the model aligns concepts across its embedding space.
The Dynamics of Knowledge Acquisition
The Early Bloom
Factual knowledge is not acquired linearly. Instead, the model experiences an "early bloom," where the majority of facts are learned very early in the pretraining process. In English, factual accuracy reaches approximately 80% within the first 50,000 steps—representing only about 12% of the total training duration.
Frequency as the Primary Engine
The strongest predictor of whether a model "knows" a fact is its frequency in the training corpus. The researchers found a massive correlation ($r = 0.93$) between subject-object co-occurrence counts and the probability of correct recall. This suggests that for the vast majority of factual knowledge, the model is performing high-level memorization of statistical co-occurrences.
Cross-Lingual Transfer and Alignment
Surprisingly Correct Low-Frequency Predictions (SCLFPs)
Despite the dominance of frequency, the model occasionally recalls facts in non-English languages that appear rarely in those specific languages. These are termed SCLFPs. The study found that these successes are driven by representational alignment: the model maps the rare non-English query to its high-frequency English counterpart.
The Role of Scripts and Entities
Cross-lingual transfer is not uniform across languages. It is significantly more effective for:
- Shared Scripts: Languages using the Latin script (like French or Spanish) show better transfer from English than those with unique scripts (like Arabic or Japanese).
- Named Entities: Proper nouns like "Paris" or "Apple" act as "anchors." Because these strings often appear identically across languages, they provide the necessary bridge for the model to align different linguistic contexts to the same underlying factual concept.
Conclusion
Factual knowledge acquisition in LLMs is characterized by rapid early learning and a heavy reliance on data frequency. However, the model's ability to align representations across languages allows it to overcome low-frequency barriers in non-English data, provided there are sufficient "anchors" like shared scripts or named entities.
Limitations
This study focuses on the English-centric OLMo-7B model. Future research is required to determine if these dynamics hold in models trained on more balanced multilingual corpora. Additionally, while the correlation between frequency and recall is strong, causal interventions—such as targeted data removal—are needed to fully confirm the mechanics of factual acquisition.
Summary updated