Measuring Social Integration Through Participation: Categorizing Organizations and Leisure Activities in the Displaced Karelians Interview Archive using LLMs

Joonatan Laato, Veera Schroderus, Jenna Kanerva, Jenni Kauppi, Virpi Lummaa, Filip Ginter

TL;DR

Researchers have developed a pipeline using Large Language Models (LLMs) to categorize over 350,000 mentions of hobbies and organizations within a massive archive of Finnish World War II evacuee interviews. By mapping these messy, real-world mentions to a structured sociological framework, the team achieved near-human performance, enabling large-scale quantitative study of social integration and its long-term effects on health and longevity.

What Problem Does This Paper Address?

Digitizing historical archives is only the first step in modern historiography. While we can now use Named Entity Recognition (NER) to pull out thousands of mentions of "fishing," "choir," or "The Martha Club," these raw strings don't tell us much about the nature of the activity.

To study social integration—how well displaced people weave themselves back into the fabric of society—sociologists need to know more than just the name of a club. They need to know: Is this a solitary or social activity? Is it physically demanding? Does it happen weekly or once a year? Manually labeling the 71,000 unique entities found in the "Displaced Karelians" archive (covering 160,000 individuals) is a Herculean task. The authors address this bottleneck by testing whether LLMs can act as "expert annotators" to categorize these entities into a multi-dimensional sociological schema.

Methods

The researchers developed a four-dimensional categorization framework:

  1. Q1 - Categories: The general type (e.g., Professional, Religious, Sports).
  2. Q2 - Group Size: Solitary vs. small group vs. large group.
  3. Q3 - Regularity: How often the activity occurs.
  4. Q4 - Physical Intensity: From stationary to intense muscular work.

To implement this at scale, they:

  • Pre-processed Hierarchies: Used an LLM (Llama-3.3) to separate roles (e.g., "Chairman") from the actual organization name.
  • Evaluated Open-Weight Models: Tested six state-of-the-art models, including Qwen2.5, DeepSeek-R1, and Mistral-Large.
  • Automated Prompt Optimization: Used "optimizer" models (OpenAI’s o3 and Anthropic’s Claude 4 Opus) to analyze errors and refine the instructions given to the labeling models.
  • Ensemble Voting: Applied a 7-prompt majority voting approach to mitigate the stochastic (random) nature of LLM outputs.

Datasets

  • Siirtokarjalaisten tie (The Path of Displaced Karelians): A 4-volume book series containing 89,339 family interviews conducted in the late 1960s.
  • The Entity Dataset: 354,302 mentions of leisure activities and organizations (71,874 unique entities).
  • Gold Standard Set: A manually annotated set of 200 entities (50 for evaluation, 150 for testing) labeled by four human experts (sociologists and ML specialists).

Key Insights and Ideas

  • The "Long Tail" of Organizations: Hobbies are highly concentrated; the top 200 hobbies account for 92.6% of all mentions. Organizations, however, have a massive "long tail," where the top 200 only cover 49% of mentions. This highlights why automated categorization is essential—you can't just manually label the "top" items and call it a day.
  • Holistic vs. Focused Prompting: Interestingly, the models performed better when asked to answer all four questions at once for an entity, rather than focusing on one question at a time. This suggests the models benefit from a "holistic" understanding of the activity's context.
  • The Cultural Knowledge Gap: The biggest failures occurred with rare, culturally specific Finnish items (like specific folk dances) or regional abbreviations. While LLMs are polyglots, they still lack the deep "common sense" history that a local expert possesses.
  • Model Overconfidence: A recurring issue was the LLM's tendency to guess a category even when the information was ambiguous, whereas human annotators were more likely to use the "Cannot be determined" label.

Key Results

  • Top Performer: Mistral-Large-Instruct-2407 emerged as the strongest model, achieving a 74.6% average F1-score on the evaluation set.
  • Near-Human Accuracy: On a simplified "coarse" label schema (e.g., collapsing "small group" and "large group" into "social"), the optimized LLM ensemble reached a 76.8% F1-score, which is 93.9% of the human baseline (81.8%).
  • Scale: The researchers successfully processed the entire dataset of 60,647 unique entities using over 424,000 LLM queries, creating a structured resource that would have taken years to produce manually.

Limitations and Future Work

The authors acknowledge that LLMs still struggle with uncertainty avoidance—they prefer to be wrong than to admit they don't know. There are also clear cultural knowledge gaps regarding Finnish-specific terminology and historical abbreviations.

Future work involves using this newly structured data to answer the "big" sociological questions: Did participation in "intense" or "social" activities actually lead to longer lives for these refugees? By linking this participation data to mortality records, the team aims to provide a quantitative look at the protective power of social integration.

Summary updated

Abstract

Digitized historical archives make it possible to study everyday social life on a large scale, but the information extracted directly from text often does not directly allow one to answer the research questions posed by historians or sociologists in a quantitative manner. We address this problem in a large collection of Finnish World War II Karelian evacuee family interviews. Prior work extracted more than 350K mentions of leisure time activities and organizational memberships from these interviews, yielding 71K unique activity and organization names -- far too many to analyze directly. We develop a categorization framework that captures key aspects of participation (the kind of activity/organization, how social it typically is, how regularly it happens, and how physically demanding it is). We annotate a gold-standard set to allow for a reliable evaluation, and then test whether large language models can apply the same schema at scale. Using a simple voting approach across multiple model runs, we find that an open-weight LLM can closely match expert judgments. Finally, we apply the method to label the 350K entities, producing a structured resource for downstream studies of social integration and related outcomes.
BibTeX
@inproceedings{laato2026measuring,
  author = {Joonatan Laato and Veera Schroderus and Jenna Kanerva and Jenni Kauppi and Virpi Lummaa and Filip Ginter},
  title = {{Measuring Social Integration Through Participation: Categorizing Organizations and Leisure Activities in the Displaced Karelians Interview Archive using LLMs}},
  booktitle = {LaTeCH-CLfL},
  year = {2026},
  doi = {10.18653/v1/2026.latechclfl-1.11},
}
Figure