Our Heritage, Our Stories: developing AI tools to link and support community-generated digital cultural heritage

Ewan David Hannaford, Viktor Schlegel, Rhiannon Lewis, Stefan Ramsden, Jenny Bunn, John Moore, Marc Alexander, Hannah Barker, Riza Batista-Navarro, Lorna M. Hughes, Goran NenadicJ. Documentation 2024 · DOI · OpenReview

Introduction

The United Kingdom's cultural landscape is enriched by a vast reservoir of community-generated digital content (CGDC), ranging from local history blogs to digitized family archives. However, this material is often "critically endangered." Unlike formal museum collections, CGDC frequently lacks standardized metadata and utilizes regional dialects that standard AI tools struggle to interpret. The Our Heritage, Our Stories (OHOS) project addresses this by building a scalable AI framework to link these fragmented histories to national archives without compromising their unique local character.

System Architecture and Pipeline

The core of the OHOS project is a multi-stage AI pipeline designed to transform unstructured community data into a searchable, interconnected web of information. This process involves several specialized labs and a dedicated "Observatory" for public interaction.

The OHOS system architecture, illustrating the data flow from community labs through the AI pipeline to the National Archives Observatory.

As shown in the architecture diagram, data flows from Archives and History Labs into a central input repository. The AI Lab Pipeline then performs Natural Language Processing (NLP) and Entity Linking before writing the results to the Observatory, where an API facilitates discovery and user interaction via the "Remixer" tool.

Methodological Innovations

To handle the specific challenges of historical and community data, the project employs several advanced techniques:

  • Named Entity Recognition (NER) & Linking: Custom models identify people, places, and niche categories (e.g., historical vessel types) and link them to global knowledge bases like Wikidata.
  • Open Relation Extraction (RE): The system models relationships dynamically (e.g., "Person X served on Ship Y") rather than relying on a fixed vocabulary, allowing for the discovery of unexpected connections.
  • Linguistic Diversity: To respect regional identities, the project treats dialects like Scots and Welsh as "low-resource languages," using optimized word embeddings and historical dictionaries to avoid erasing unique linguistic context.
  • Few-Shot Learning: Machine learning models are trained to recognize new entities using minimal human-labeled examples, significantly increasing the scalability of the pipeline across diverse collections.

Strategic Insights: "History from Below"

The project advocates for a "post-custodial" archiving philosophy. Rather than the national archive "owning" community data, it provides the infrastructure to make it visible and linkable. A key technical insight is the prioritization of high recall over high precision; for historical research, finding all possible connections is often more valuable than avoiding all false positives, provided confidence scores are available for filtering.

Conclusion and Future Work

By integrating community voices into the national record, OHOS ensures that "citizen history" is preserved alongside official narratives. While the pipeline has been successfully demonstrated on datasets like the Morrab Library and The National Archives' 'Discovery' catalogue, future work focuses on the long-term sustainability of this infrastructure and refining models to handle increasingly niche historical topics with minimal human intervention.

Summary updated

Abstract

Purpose Community-generated digital content (CGDC) is one of the UK’s prime cultural assets. However, CGDC is currently “critically endangered” (Digital Preservation Coalition, 2021) due to technological and organisational barriers and has proven resistant to traditional methods of linking and integration. The challenge of integrating CGDC into larger archives has effectively silenced diverse community voices within our national collection. Our Heritage, Our Stories (OHOS), funded by the UK’s AHRC programme Towards a National Collection, responds to these urgent challenges by bringing together cutting-edge approaches from cultural heritage, humanities and computer science. Design/methodology/approach Existing solutions to CGDC integration, involving bespoke interventionist activities, are expensive, time-consuming and unsustainable at scale, while unsophisticated computational integration erases the meaning and purpose of both CGDC and its creators. Using innovative multidisciplinary methods, AI tools and a co-design process, previously unfindable and unlinkable CGDC will be made discoverable in our virtual national collection. Findings There currently exists a range of disconnected, fragile and under-represented community-generated heritage which is at increasing risk of loss. Therefore, OHOS will work to ensure the survival and preservation of these nationally important resources, for the future and for our shared national collection. Originality/value As we dissolve barriers to create meaningful new links across CGDC collections and develop new methods of engagement, OHOS will also make this content accessible to new and diverse audiences. This will facilitate a wealth of fresh research while also embedding new strategies for future management of CGDC into heritage practice and training and fostering newly enriching, robust connections between communities and archival institutions.
BibTeX
@article{hannaford2024our,
  author = {Ewan David Hannaford and Viktor Schlegel and Rhiannon Lewis and Stefan Ramsden and Jenny Bunn and John Moore and Marc Alexander and Hannah Barker and Riza Batista-Navarro and Lorna M. Hughes and Goran Nenadic},
  title = {{Our Heritage, Our Stories: developing AI tools to link and support community-generated digital cultural heritage}},
  journal = {J. Documentation},
  year = {2024},
  doi = {10.1108/jd-03-2024-0057},
}
Figure