Introduction
The United Kingdom's cultural landscape is enriched by a vast reservoir of community-generated digital content (CGDC), ranging from local history blogs to digitized family archives. However, this material is often "critically endangered." Unlike formal museum collections, CGDC frequently lacks standardized metadata and utilizes regional dialects that standard AI tools struggle to interpret. The Our Heritage, Our Stories (OHOS) project addresses this by building a scalable AI framework to link these fragmented histories to national archives without compromising their unique local character.
System Architecture and Pipeline
The core of the OHOS project is a multi-stage AI pipeline designed to transform unstructured community data into a searchable, interconnected web of information. This process involves several specialized labs and a dedicated "Observatory" for public interaction.
As shown in the architecture diagram, data flows from Archives and History Labs into a central input repository. The AI Lab Pipeline then performs Natural Language Processing (NLP) and Entity Linking before writing the results to the Observatory, where an API facilitates discovery and user interaction via the "Remixer" tool.
Methodological Innovations
To handle the specific challenges of historical and community data, the project employs several advanced techniques:
- Named Entity Recognition (NER) & Linking: Custom models identify people, places, and niche categories (e.g., historical vessel types) and link them to global knowledge bases like Wikidata.
- Open Relation Extraction (RE): The system models relationships dynamically (e.g., "Person X served on Ship Y") rather than relying on a fixed vocabulary, allowing for the discovery of unexpected connections.
- Linguistic Diversity: To respect regional identities, the project treats dialects like Scots and Welsh as "low-resource languages," using optimized word embeddings and historical dictionaries to avoid erasing unique linguistic context.
- Few-Shot Learning: Machine learning models are trained to recognize new entities using minimal human-labeled examples, significantly increasing the scalability of the pipeline across diverse collections.
Strategic Insights: "History from Below"
The project advocates for a "post-custodial" archiving philosophy. Rather than the national archive "owning" community data, it provides the infrastructure to make it visible and linkable. A key technical insight is the prioritization of high recall over high precision; for historical research, finding all possible connections is often more valuable than avoiding all false positives, provided confidence scores are available for filtering.
Conclusion and Future Work
By integrating community voices into the national record, OHOS ensures that "citizen history" is preserved alongside official narratives. While the pipeline has been successfully demonstrated on datasets like the Morrab Library and The National Archives' 'Discovery' catalogue, future work focuses on the long-term sustainability of this infrastructure and refining models to handle increasingly niche historical topics with minimal human intervention.
Summary updated