Extracting Social Connections from Finnish Karelian Refugee Interviews Using LLMs
Introduction
Researchers have successfully leveraged Large Language Models (LLMs) to extract structured social connection data from a massive historical archive: nearly 90,000 interviews of Finnish refugees displaced after World War II. This study demonstrates that state-of-the-art models like GPT-4 and Llama-3-70B can rival human performance in complex information extraction tasks for non-English languages. Furthermore, the research shows that these "heavyweight" models can effectively generate training data to create smaller, specialized models (such as FinBERT) that achieve high accuracy with significantly lower computational requirements.
The Challenge of Historical Text Extraction
Digital humanities researchers often encounter a "mountain of text"—millions of words of valuable historical data that lack a structured format for analysis. Traditionally, extracting specific entities, such as hobbies or organizational memberships, required thousands of hours of manual annotation to train machine learning models. This study addresses three core challenges: handling Finnish (a lower-resource language) in a historical context, correctly attributing entities (e.g., distinguishing between a husband's and wife's hobbies in messy text), and ensuring the process is economically and computationally scalable for nearly 90,000 interviews.
Methodology
The study compared several Information Extraction (IE) strategies:
- Zero-Shot Prompting: Directly asking various LLMs (including GPT-4, Llama-3, Mistral, and Qwen) to extract data without specific prior training.
- Prompt Engineering: Testing the impact of prompt language (Finnish vs. English) and the effectiveness of "batching" multiple interviews into a single prompt.
- Model Distillation: Using GPT-4 to generate "silver" training data for 30,000 interviews, which was then used to fine-tune FinBERT—a smaller, encoder-only model optimized for the Finnish language.
Datasets and Evaluation
The primary dataset, Siirtokarjalaisten tie (The Journey of Displaced Karelians), consists of 89,339 family interviews conducted between 1968 and 1970. To evaluate model performance, the authors created a "gold standard" benchmark by manually annotating 400 interviews. Human annotators achieved an 83.4% agreement rate, establishing a baseline for model accuracy.
Key Results and Insights
The results highlight a significant leap in the capabilities of open-source models. Llama-3-70B-Instruct achieved an 87.7% F-score, nearly matching the 88.8% F-score of the proprietary GPT-4. This parity suggests that researchers can now achieve state-of-the-art results using local infrastructure, which is crucial for data privacy and reproducibility.
Other key findings include:
- Prompt Language: GPT-4's performance improved by 3 percentage points when prompted in Finnish, whereas GPT-3.5's performance declined.
- Batching Inefficiency: Processing multiple interviews in one prompt significantly reduced accuracy, as LLMs perform best when target information is at the beginning of the context window.
- Success of Distillation: The fine-tuned FinBERT model reached an 86.3% F-score. This smaller model is efficient enough to run on standard hardware while maintaining high accuracy.
Conclusion and Future Directions
While the current pipeline successfully extracts text entities, future work will focus on "entity grounding"—mapping extracted names (e.g., "Martha club") to standardized database entries. This structured data will eventually enable deep socio-historical analysis, exploring how social connections influenced the long-term integration and lifespans of the Karelian refugee population. Integrating this data with digitized church and census records will further build a comprehensive historical profile of this population.
Summary updated