Art Speaks Maths, Maths Speaks Art

Ninetta Leone, Simone Parisotto, Kasia Targonska-Hadzibabic, Spike Bucklow, Alessandro Launaro, Suzanne Reynolds, Carola-Bibiane Schönlieb

TL;DR

The Mathematics for Applications in Cultural Heritage (MACH) project bridges the gap between advanced mathematics and the arts by developing specialized, user-friendly toolkits for art conservation and archaeology. By fostering an iterative dialogue where "art speaks maths" to define problems and "maths speaks art" to provide solutions, the team is enabling non-invasive restoration and systematic classification of historical artifacts.

What Problem Does This Paper Address?

Interdisciplinary collaboration often suffers from a fundamental communication gap. In the realm of cultural heritage, the arts and humanities community possesses vast amounts of data but lacks the sophisticated tools to process it. Conversely, mathematicians possess powerful analytical tools but often lack the domain-specific knowledge to identify meaningful questions or communicate results in a non-technical language.

Furthermore, mathematical research typically relies on an objective "ground truth" to validate models. In art conservation, however, ground truth is often subjective—as the authors note, art historians "know it when they see it." This paper describes how the MACH team tackles these challenges across three specific domains: painting conservation, Roman pottery classification, and the restoration of illuminated manuscripts.

Methods

The researchers employ a diverse array of mathematical and computational techniques tailored to each project:

  • Painting Cross-Sections: To analyze paint layers, the team uses deep neural networks to isolate cross-sections from inconsistent backgrounds. They then apply content-based image retrieval (CBIR) with k-nearest neighbors, utilizing Gabor filters for texture and spatial distribution moments for color similarity.
  • Pottery Classification: To move beyond inconsistent manual catalogs, the team uses sparse autoencoders to extract features into a latent subspace. These features then undergo hierarchical clustering to create a systematic, tree-like classification of pottery shapes, even when dealing with fragmented remains.
  • Manuscript Restoration: The team utilizes virtual inpainting to restore images non-invasively. They employ diffusion-type partial differential equations (PDEs) for small scratches and exemplar-based approaches for larger holes. For multi-spectral data fusion, they developed an ad-hoc approach based on a variational version of the drift-diffusion osmosis equation, which filters noise from one channel (like infra-red) while encoding structural information into the visible channel.
The scope of the MACH project across three domains: (a) painting cross-sections, (b) pottery shapes, and (c) manuscript digitisation.

Datasets

The project leverages several significant archives:

  • Paintings: Over 10,000 digital images of cross-sections from the Hamilton Kerr Institute, collected over 40 years.
  • Pottery: A dataset of 5,000+ published profiles of Roman commonware pottery.
  • Manuscripts: More than 35,000 digital images from 2,000 manuscripts held at the Fitzwilliam Museum and various Cambridge Colleges.

Key Insights and Ideas

  • The Toolkit as a Mediator: Perhaps the most profound insight is that the software toolkits are not just the final product; they are "indispensable mediators." By providing a visual interface for mathematical operations, they allow art experts to provide the continuous feedback necessary to drive mathematical development in the absence of objective ground truth.
  • Standardization via Deep Learning: A major hurdle in cultural heritage is the "dirty" nature of legacy data. Using deep learning to automate the pre-processing and standardization of 40 years of varied archival data was essential for systematic analysis.
  • Iterative Symmetry: The paper emphasizes that for "maths to speak art" (providing useful tools), "art must first speak maths" (defining the data and the questions). This iterative loop ensures the mathematical engines are optimized for actual end-user needs rather than theoretical elegance alone.

Key Results

The primary output of this work is the development of three distinct software toolkits, currently under development, which allow users to perform complex analyses intuitively.

Examples of the MACH toolkits for cross-section analysis, virtual restoration, and pottery clustering.

Qualitatively, the project has demonstrated the ability to reveal non-obvious connections between painting techniques (such as "optical blues" created by scattering) and to provide a coherent chronological and distributional map of Roman pottery. In the manuscript project, the tools have enabled the virtual "cleaning" of overpainted sections, such as restoring the original appearance of a prayer book belonging to Yolande of Anjou.

Limitations and Future Work

The authors acknowledge that their current toolkits are still under development and require broader feedback from the wider arts and humanities community to reach their full potential. The next phase involves releasing the software publicly to invite this wider critique.

Looking forward, the team hopes that this collaboration will not only solve existing problems but inspire mathematicians to develop entirely new algorithms based on artistic challenges that have not yet been conceptualized. The ultimate goal is a sustainable ecosystem where mathematical research and cultural preservation continuously inform one another.

Summary updated

Abstract

Our interdisciplinary team Mathematics for Applications in Cultural Heritage (MACH) aims to use mathematical research for the benefit of the arts and humanities. Our ultimate goal is to create user-friendly software toolkits for artists, art conservators and archaeologists. In order for their underlying mathematical engines and functionality to be optimised for the needs of the end users, we pursue an iterative approach based on a continuous communication between the mathematicians and the cultural-heritage members of our team. Our paper illustrates how maths can speak art, but only if first art speaks maths.
BibTeX
@misc{leone2020art,
  author = {Ninetta Leone and Simone Parisotto and Kasia Targonska-Hadzibabic and Spike Bucklow and Alessandro Launaro and Suzanne Reynolds and Carola-Bibiane Schönlieb},
  title = {{Art Speaks Maths, Maths Speaks Art}},
  year = {2020},
  eprint = {2007.08886},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2007.08886},
}
Figure