Extracting Social Connections from Finnish Karelian Refugee Interviews Using LLMs

Joonatan Laato, Jenna Kanerva, John Loehr, Virpi Lummaa, Filip GinterarXiv 2025 · DOI · arXiv

Extracting Social Connections from Finnish Karelian Refugee Interviews Using LLMs

Introduction

Researchers have successfully leveraged Large Language Models (LLMs) to extract structured social connection data from a massive historical archive: nearly 90,000 interviews of Finnish refugees displaced after World War II. This study demonstrates that state-of-the-art models like GPT-4 and Llama-3-70B can rival human performance in complex information extraction tasks for non-English languages. Furthermore, the research shows that these "heavyweight" models can effectively generate training data to create smaller, specialized models (such as FinBERT) that achieve high accuracy with significantly lower computational requirements.

The Challenge of Historical Text Extraction

Digital humanities researchers often encounter a "mountain of text"—millions of words of valuable historical data that lack a structured format for analysis. Traditionally, extracting specific entities, such as hobbies or organizational memberships, required thousands of hours of manual annotation to train machine learning models. This study addresses three core challenges: handling Finnish (a lower-resource language) in a historical context, correctly attributing entities (e.g., distinguishing between a husband's and wife's hobbies in messy text), and ensuring the process is economically and computationally scalable for nearly 90,000 interviews.

Methodology

The study compared several Information Extraction (IE) strategies:

  1. Zero-Shot Prompting: Directly asking various LLMs (including GPT-4, Llama-3, Mistral, and Qwen) to extract data without specific prior training.
  2. Prompt Engineering: Testing the impact of prompt language (Finnish vs. English) and the effectiveness of "batching" multiple interviews into a single prompt.
  3. Model Distillation: Using GPT-4 to generate "silver" training data for 30,000 interviews, which was then used to fine-tune FinBERT—a smaller, encoder-only model optimized for the Finnish language.
Prompt instructions for extracting structured data from interview text.

Datasets and Evaluation

The primary dataset, Siirtokarjalaisten tie (The Journey of Displaced Karelians), consists of 89,339 family interviews conducted between 1968 and 1970. To evaluate model performance, the authors created a "gold standard" benchmark by manually annotating 400 interviews. Human annotators achieved an 83.4% agreement rate, establishing a baseline for model accuracy.

Key Results and Insights

The results highlight a significant leap in the capabilities of open-source models. Llama-3-70B-Instruct achieved an 87.7% F-score, nearly matching the 88.8% F-score of the proprietary GPT-4. This parity suggests that researchers can now achieve state-of-the-art results using local infrastructure, which is crucial for data privacy and reproducibility.

Performance comparison of various open-source LLMs on the extraction task.

Other key findings include:

  • Prompt Language: GPT-4's performance improved by 3 percentage points when prompted in Finnish, whereas GPT-3.5's performance declined.
  • Batching Inefficiency: Processing multiple interviews in one prompt significantly reduced accuracy, as LLMs perform best when target information is at the beginning of the context window.
  • Success of Distillation: The fine-tuned FinBERT model reached an 86.3% F-score. This smaller model is efficient enough to run on standard hardware while maintaining high accuracy.

Conclusion and Future Directions

While the current pipeline successfully extracts text entities, future work will focus on "entity grounding"—mapping extracted names (e.g., "Martha club") to standardized database entries. This structured data will eventually enable deep socio-historical analysis, exploring how social connections influenced the long-term integration and lifespans of the Karelian refugee population. Integrating this data with digitized church and census records will further build a comprehensive historical profile of this population.

Summary updated

Abstract

We performed a zero-shot information extraction study on a historical collection of 89,339 brief Finnish-language interviews of refugee families relocated post-WWII from Finnish Eastern Karelia. Our research objective is two-fold. First, we aim to extract social organizations and hobbies from the free text of the interviews, separately for each family member. These can act as a proxy variable indicating the degree of social integration of refugees in their new environment. Second, we aim to evaluate several alternative ways to approach this task, comparing a number of generative models and a supervised learning approach, to gain a broader insight into the relative merits of these different approaches and their applicability in similar studies. We find that the best generative model (GPT-4) is roughly on par with human performance, at an F-score of 88.8%. Interestingly, the best open generative model (Llama-3-70B-Instruct) reaches almost the same performance, at 87.7% F-score, demonstrating that open models are becoming a viable alternative for some practical tasks even on non-English data. Additionally, we test a supervised learning alternative, where we fine-tune a Finnish BERT model (FinBERT) using GPT-4 generated training data. By this method, we achieved an F-score of 84.1% already with 6K interviews up to an F-score of 86.3% with 30k interviews. Such an approach would be particularly appealing in cases where the computational resources are limited, or there is a substantial mass of data to process.
BibTeX
@misc{laato2025extracting,
  author = {Joonatan Laato and Jenna Kanerva and John Loehr and Virpi Lummaa and Filip Ginter},
  title = {{Extracting Social Connections from Finnish Karelian Refugee Interviews Using LLMs}},
  year = {2025},
  eprint = {2502.13566},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2502.13566},
}
Figure