Introduction
While Large Language Models (LLMs) excel at general reasoning, they often struggle to internalize deep, "buried" insights from specialized fields like medicine or finance. Conventional solutions like Retrieval-Augmented Generation (RAG) act as a "lookup" strategy but fail to help the model truly internalize patterns, statistical distributions, or probabilistic relationships spread across vast document sets. This paper investigates whether Continual Pre-Training (CPT) using LoRA (Low-Rank Adaptation) can bridge this gap, moving beyond surface-level facts to learn declarative, statistical, and probabilistic insights.
Methodology
The researchers evaluated three LLaMA variants (LLaMA-3.2 1B, 3B, and LLaMA-3.1 8B) using LoRA-based continual pre-training. The core of the experiment involved comparing training on raw, original documents versus structured, "denoised" documents.
The approach consisted of:
- Insight Mining: Utilizing GPT-4o mini to extract information "triples" (subject-relation-object) from raw text, which were then manually filtered to create a high-quality ground truth.
- Document Simplification: Comparing training on original prose versus simplified documents where only essential triples were retained.
- Evaluation Framework: A benchmark was developed to test the models' ability to predict missing objects in declarative and statistical queries and to estimate conditional probabilities.
Experimental Setup and Results
The study focused on two domains: Hallmarks of Cancer (medical) and Buster (financial). The benchmark included 500 queries per insight type, sampled for uniform difficulty.
Key Findings
- The "Noise" Barrier: Training on raw documents yielded only marginal improvements. The natural language "fluff" appears to distract the model during the LoRA update process.
- Structure as a Force Multiplier: Converting prose into structured triples acted as a powerful denoiser. On the Hallmarks of Cancer dataset, LLaMA-3.1 8B achieved a 99.0% Exact Match for declarative insights and 82.0% Recall@10 for statistical insights after training on simplified triples.
- The Probabilistic Ceiling: Models struggled significantly with probabilistic insights regardless of data format. Mean Absolute Error (MAE) remained high (0.14–0.16), suggesting LoRA's limitations in updating internal "world logic" regarding uncertainty.
- Scaling Effects: Larger models (8B) consistently outperformed smaller ones (1B), indicating that "insight capacity" scales with parameter count even under low-rank adaptation.
Discussion and Future Work
The results suggest that while LoRA is effective for "memorizing" facts and simple aggregations when data is structured, it may be fundamentally limited for complex probabilistic reasoning. The authors hypothesize that LoRA "learns less and forgets less" than full fine-tuning, which might be a bottleneck for deep insight acquisition.
Future research will explore:
- Full Fine-Tuning: Investigating if removing the LoRA constraint allows models to break the probabilistic ceiling.
- Complex Reasoning: Moving beyond atomic triples to multi-step reasoning and interconnected insights.
- Ambiguity Handling: Developing methods to teach LLMs context-dependent knowledge that isn't universally true outside specific domain datasets.
Summary updated