Heritage Iconographic Content Structuring: from Automatic Linking to Visual Validation

Emile Blettery, Valérie Gouet-BrunetJournal on Computing and Cultural Heritage 2024 · DOI

TL;DR

This paper introduces a novel framework for structuring vast, interconnected iconographic heritage collections. By combining advanced content-based image retrieval with an interactive 3D visualization platform, they enable semi-automatic correction and refinement of links, significantly improving data organization and accessibility. This approach is crucial for unlocking the full potential of digitized cultural heritage.

What Problem Does This Paper Address?

Digitized iconographic heritage collections, often spread across multiple institutions, suffer from poor interlinking and organization. This "siloed" nature makes it difficult to discover, analyze, and cross-reference valuable historical content. Traditional methods rely heavily on manual curation and metadata, which are often inconsistent or incomplete across different collections. The inherent visual heterogeneity of heritage images (paintings, photographs, maps, etc.) further complicates automatic linking. This paper tackles the challenge of creating a unified, structured representation of these collections, bridging gaps between them and enabling more effective analysis.

Methods

The authors propose a multi-stage framework:

  1. Automatic Linking: They evaluate and adapt content-based image retrieval (CBIR) techniques, focusing on image descriptors and re-ranking strategies that can handle the visual variability of heritage content. They specifically highlight the effectiveness of local descriptors (like How+ASMK) and diffusion-based re-ranking for cross-collection linking.
  2. Graph-Based Representation: The structured dataset is represented as a multi-edge graph, incorporating visual similarity links, spatial similarity links, and location links. This allows for a comprehensive understanding of relationships between images and locations.
  3. 3D Immersive Visualization Platform (GraphXR): A custom platform is developed to visualize this graph structure. It offers interactive tools for exploring the dataset, identifying potential errors, and facilitating manual interventions.
  4. Iterative Semi-Automatic Structuring: This core contribution combines automatic linking with manual correction. Experts use the visualization platform to identify and correct errors, add new links, or refine existing ones. These manual interventions are then propagated through the graph, and the process can be iterated to progressively improve the structure.

Datasets

The primary dataset used for evaluation is DBquery+dist, a collection of approximately 1,637 images of outdoor areas in Paris from 1915-2015. This dataset is sourced from eight different providers, including Parisian institutions and the French mapping agency. To challenge the retrieval systems, an additional 8,197 distractor images were included, bringing the total to 9,834 images. The dataset is characterized by high visual variability, diverse representation scales, and changes in depicted scenes over time.

Key Insights and Ideas

  • Visual Heterogeneity is Key: The paper strongly emphasizes that standard CBIR methods often struggle with the diverse visual styles and quality of heritage images. Local descriptors and robust re-ranking strategies are crucial for effective linking.
  • Diffusion for Cross-Collection Linking: Diffusion-based re-ranking methods show significant promise for bridging gaps between different collections by leveraging a broader neighborhood of similar images, even those with indirect connections.
  • Visualization as a Tool for Correction: A 3D immersive visualization platform is not just for browsing but is essential for identifying structural flaws and guiding manual corrections. Visual clues like "highly central nodes" and "cross-community links" are powerful indicators of potential errors.
  • Semi-Automatic is Better Than Purely Automatic: The iterative process of combining automatic linking with expert manual intervention significantly outperforms purely automatic methods, especially for complex or ambiguous cases. This hybrid approach leverages the strengths of both machine and human intelligence.
  • Provider Entropy Matters: The performance of diffusion-based re-ranking is significantly boosted when the initial retrieved results contain images from a diverse set of providers, suggesting that a mix of sources helps the diffusion process explore the broader graph structure more effectively.

Key Results

  • Improved mAP Scores: The proposed semi-automatic process, particularly when building on strong automatic linking and diffusion, achieved significant improvements in mAP scores. For instance, an iterative process involving manual interventions on the 5th-10th links resulted in an mAP score of 64.21%, a notable increase from the initial automatic linking score of 61.77%.
  • Efficiency of Semi-Automatic Approach: The paper demonstrates that the semi-automatic approach offers a favorable trade-off between computation time and mAP improvement compared to exhaustive automatic re-ranking. For example, 1 hour of automatic processing could not achieve the same level of improvement as a few hours of the semi-automatic process.
  • Effectiveness of Manual Interventions: Manual interventions, such as deleting incorrect links or creating new spatial/expert similarity links, consistently improved mAP scores, especially when applied to deeper, more uncertain links in the graph.

Limitations and Future Work

  • Scalability of Visualization: The authors acknowledge that the visualization platform (GraphXR) has limitations in handling extremely large graphs (recommending a maximum of 10,000 nodes). They suggest strategies like displaying subsets of links or focusing on specific areas to mitigate this.
  • Retraining of Descriptors: The current evaluation uses pre-trained image descriptors. The authors suggest that retraining these descriptors on dedicated iconographic heritage datasets could further improve performance.
  • Integration of Semantic Information: While the current framework focuses on visual and spatial similarities, the authors propose that incorporating semantic information extracted from metadata or automatic classification could further enhance the structuring and visualization.
  • User Interface Tailoring: For broader adoption, more tailored and conventional search interfaces might be needed to exploit the structured data in user-friendly ways for specific applications (e.g., digital tourism, urban planning).

Summary updated

Abstract

This article presents a global framework dedicated to the structuring of iconographic heritage collections. To alleviate the poor interlinking both between collections and contents, a first step of automatic linking exploiting content-based image retrieval approaches is evaluated and adapted to the visual variability of such heritage contents. To ensure understanding and analysis of the contents in a structured fashion, a 3D immersive web platform is also introduced alongside visual-based analysis tools. Finally, by exploiting both automatic linking and manual interventions in the visualization platform, an iterative, semi-automatic structuring pipeline is proposed to solve difficult cases missed by automatic structuring, and then improve structuring optimally. Here, we demonstrate the potential of the proposal on the geographic iconographic heritage of Paris, with a dataset of 10k images belonging to several institutions, thus poorly connected nor organized globally.
BibTeX
@article{blettery2024heritage,
  author = {Emile Blettery and Valérie Gouet-Brunet},
  title = {{Heritage Iconographic Content Structuring: from Automatic Linking to Visual Validation}},
  journal = {Journal on Computing and Cultural Heritage},
  year = {2024},
  doi = {10.1145/3666007},
}
Figure