Measuring Social Integration Through Participation: Categorizing Organizations and Leisure Activities in the Displaced Karelians Interview Archive using LLMs

Joonatan Laato, Veera I. Schroderus, Jenna Kanerva, Jenni Kauppi, Virpi Lummaa, Filip Ginter

Abstract

Digitized historical archives make it possible to study everyday social life on a large scale, but the information extracted directly from text often does not directly allow one to answer the research questions posed by historians or sociologists in a quantitative manner. We address this problem in a large collection of Finnish World War II Karelian evacuee family interviews. Prior work extracted more than 350K mentions of leisure time activities and organizational memberships from these interviews, yielding 71K unique activity and organization names -- far too many to analyze directly. We develop a categorization framework that captures key aspects of participation (the kind of activity/organization, how social it typically is, how regularly it happens, and how physically demanding it is). We annotate a gold-standard set to allow for a reliable evaluation, and then test whether large language models can apply the same schema at scale. Using a simple voting approach across multiple model runs, we find that an open-weight LLM can closely match expert judgments. Finally, we apply the method to label the 350K entities, producing a structured resource for downstream studies of social integration and related outcomes.

Introduction

Digitizing historical archives is only the first step in modern historiography. While Named Entity Recognition (NER) can extract thousands of mentions of "fishing," "choir," or "The Martha Club," these raw strings lack the semantic depth required for sociological analysis. To study social integration—how displaced populations reintegrate into society—researchers need to understand the nature of these activities. Is a specific hobby solitary or social? Is an organization's activity physically demanding or stationary?

Manually labeling the 71,000 unique entities found in the "Displaced Karelians" archive, which covers 160,000 individuals, is a Herculean task. This paper explores the use of Large Language Models (LLMs) as "expert annotators" to categorize these entities into a multi-dimensional sociological framework.

The Categorization Framework

The researchers developed a four-dimensional schema to capture the essential characteristics of social and leisure participation:

The four-dimensional categorization framework used to classify activities and organizations.
  1. Q1 - Categories: The general domain (e.g., Professional, Religious, Sports).
  2. Q2 - Group Size: Whether the activity is typically performed alone, in small groups, or in large groups.
  3. Q3 - Regularity: The frequency of the activity (Regular, Occasional, Continuous, etc.).
  4. Q4 - Physical Intensity: The level of physical exertion required.

Methodology

The pipeline involved several stages to ensure high-quality labeling:

  • Pre-processing: Llama-3.3 was used to disentangle roles (e.g., "Chairman") from the organization names themselves.
  • Model Selection: Six state-of-the-art open-weight and proprietary models were evaluated, including Qwen2.5, DeepSeek-R1, and Mistral-Large.
  • Prompt Optimization: "Optimizer" models (OpenAI’s o3 and Anthropic’s Claude 4 Opus) refined the instructions by analyzing errors in initial runs.
  • Ensemble Voting: A 7-prompt majority voting approach was used to stabilize the stochastic outputs of the LLMs.

Results and Insights

The evaluation against a manually annotated "Gold Standard" set of 200 entities revealed that LLMs can achieve near-human performance in this complex classification task.

Average F1-scores for various LLMs on the activity categorization task.

Mistral-Large-Instruct-2407 was the top performer, with an average F1-score of 74.6%. When using a simplified "coarse" label schema (e.g., merging group size categories into a binary "social" vs. "solitary"), the optimized ensemble reached a 76.8% F1-score, representing 93.9% of the human expert baseline (81.8%).

The "Long Tail" Challenge

A key finding was the difference in distribution between hobbies and organizations. While the top 200 hobbies account for over 92% of mentions, the top 200 organizations cover less than half (49%). This "long tail" of unique organizations underscores the necessity of automated categorization; manual efforts would miss a significant portion of the social landscape.

Model Behavior and Limitations

The study noted that models performed better with "holistic" prompting—answering all four questions simultaneously—suggesting they benefit from the shared context of the entity. However, challenges remain. LLMs struggled with rare Finnish folk traditions or regional abbreviations. Furthermore, unlike human experts, LLMs often guessed a category rather than admitting information was insufficient ("Cannot be determined").

Conclusion

By processing over 60,000 unique entities through 424,000 LLM queries, this research has created a structured resource for historical sociology. Future work will link this participation data to mortality records to investigate whether specific types of social integration (e.g., "intense" vs. "stationary" or "social" vs. "solitary") correlate with long-term health and longevity among displaced populations.

Summary updated

Abstract

Digitized historical archives make it possible to study everyday social life on a large scale, but the information extracted directly from text often does not directly allow one to answer the research questions posed by historians or sociologists in a quantitative manner. We address this problem in a large collection of Finnish World War II Karelian evacuee family interviews. Prior work extracted more than 350K mentions of leisure time activities and organizational memberships from these interviews, yielding 71K unique activity and organization names -- far too many to analyze directly. We develop a categorization framework that captures key aspects of participation (the kind of activity/organization, how social it typically is, how regularly it happens, and how physically demanding it is). We annotate a gold-standard set to allow for a reliable evaluation, and then test whether large language models can apply the same schema at scale. Using a simple voting approach across multiple model runs, we find that an open-weight LLM can closely match expert judgments. Finally, we apply the method to label the 350K entities, producing a structured resource for downstream studies of social integration and related outcomes.
BibTeX
@inproceedings{laato2026measuring,
  author = {Joonatan Laato and Veera I. Schroderus and Jenna Kanerva and Jenni Kauppi and Virpi Lummaa and Filip Ginter},
  title = {{Measuring Social Integration Through Participation: Categorizing Organizations and Leisure Activities in the Displaced Karelians Interview Archive using LLMs}},
  booktitle = {LaTeCH-CLfL},
  year = {2026},
  doi = {10.18653/v1/2026.latechclfl-1.11},
}
Figure