Learning Beyond the Surface: How Far Can Continual Pre-Training with LoRA Enhance LLMs' Domain-Specific Insight Learning?

Pouya Pezeshkpour, Estevam HruschkaarXiv 2025 · DOI · arXiv

Introduction

While Large Language Models (LLMs) excel at general reasoning, they often struggle to internalize deep, "buried" insights from specialized fields like medicine or finance. Conventional solutions like Retrieval-Augmented Generation (RAG) act as a "lookup" strategy but fail to help the model truly internalize patterns, statistical distributions, or probabilistic relationships spread across vast document sets. This paper investigates whether Continual Pre-Training (CPT) using LoRA (Low-Rank Adaptation) can bridge this gap, moving beyond surface-level facts to learn declarative, statistical, and probabilistic insights.

Methodology

The researchers evaluated three LLaMA variants (LLaMA-3.2 1B, 3B, and LLaMA-3.1 8B) using LoRA-based continual pre-training. The core of the experiment involved comparing training on raw, original documents versus structured, "denoised" documents.

Overview of the Continual Pre-training and Insight Mining pipeline.

The approach consisted of:

  • Insight Mining: Utilizing GPT-4o mini to extract information "triples" (subject-relation-object) from raw text, which were then manually filtered to create a high-quality ground truth.
  • Document Simplification: Comparing training on original prose versus simplified documents where only essential triples were retained.
  • Evaluation Framework: A benchmark was developed to test the models' ability to predict missing objects in declarative and statistical queries and to estimate conditional probabilities.

Experimental Setup and Results

The study focused on two domains: Hallmarks of Cancer (medical) and Buster (financial). The benchmark included 500 queries per insight type, sampled for uniform difficulty.

Key Findings

  • The "Noise" Barrier: Training on raw documents yielded only marginal improvements. The natural language "fluff" appears to distract the model during the LoRA update process.
  • Structure as a Force Multiplier: Converting prose into structured triples acted as a powerful denoiser. On the Hallmarks of Cancer dataset, LLaMA-3.1 8B achieved a 99.0% Exact Match for declarative insights and 82.0% Recall@10 for statistical insights after training on simplified triples.
  • The Probabilistic Ceiling: Models struggled significantly with probabilistic insights regardless of data format. Mean Absolute Error (MAE) remained high (0.14–0.16), suggesting LoRA's limitations in updating internal "world logic" regarding uncertainty.
  • Scaling Effects: Larger models (8B) consistently outperformed smaller ones (1B), indicating that "insight capacity" scales with parameter count even under low-rank adaptation.

Discussion and Future Work

The results suggest that while LoRA is effective for "memorizing" facts and simple aggregations when data is structured, it may be fundamentally limited for complex probabilistic reasoning. The authors hypothesize that LoRA "learns less and forgets less" than full fine-tuning, which might be a bottleneck for deep insight acquisition.

Future research will explore:

  • Full Fine-Tuning: Investigating if removing the LoRA constraint allows models to break the probabilistic ceiling.
  • Complex Reasoning: Moving beyond atomic triples to multi-step reasoning and interconnected insights.
  • Ambiguity Handling: Developing methods to teach LLMs context-dependent knowledge that isn't universally true outside specific domain datasets.

Summary updated

Abstract

Large Language Models (LLMs) have demonstrated remarkable performance on various tasks, yet their ability to extract and internalize deeper insights from domain-specific datasets remains underexplored. In this study, we investigate how continual pre-training can enhance LLMs' capacity for insight learning across three distinct forms: declarative, statistical, and probabilistic insights. Focusing on two critical domains: medicine and finance, we employ LoRA to train LLMs on two existing datasets. To evaluate each insight type, we create benchmarks to measure how well continual pre-training helps models go beyond surface-level knowledge. We also assess the impact of document modification on capturing insights. The results show that, while continual pre-training on original documents has a marginal effect, modifying documents to retain only essential information significantly enhances the insight-learning capabilities of LLMs.
BibTeX
@misc{pezeshkpour2025learning,
  author = {Pouya Pezeshkpour and Estevam Hruschka},
  title = {{Learning Beyond the Surface: How Far Can Continual Pre-Training with LoRA Enhance LLMs' Domain-Specific Insight Learning?}},
  year = {2025},
  eprint = {2501.17840},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2501.17840},
}
Figure