Art Speaks Maths, Maths Speaks Art.

Ninetta Leone, Simone Parisotto, Katarzyna Targonska-Hadzibabic, Spike Bucklow, Alessandro Launaro, Suzanne Reynolds, Carola‐Bibiane Schönlieb

Introduction

In the traditional world of art conservation and archaeology, expertise is often synonymous with "connoisseurship"—a subjective, experience-based intuition developed over decades. However, as the volume of digital data in cultural heritage grows, there is an increasing need for objective, reproducible methods of analysis. The Mathematics for Applications in Cultural Heritage (MACH) project was established to bridge this gap, creating a bidirectional dialogue between mathematicians and art historians to develop specialized software toolkits that transform subjective observations into data-driven insights.

The Three Pillars of the MACH Project

The MACH project addresses the "language barrier" between the arts and sciences by focusing on three distinct domains, each presenting unique computational challenges: painting analysis, archaeological classification, and virtual manuscript restoration.

The scope of the MACH project across three domains: paintings, pottery, and manuscripts.

1. Computer Vision for Painting Analysis

Analyzing the physical structure of a painting often involves examining microscopic cross-sections of paint layers. With archives like the Hamilton Kerr Institute containing over 10,000 such images, manual analysis is no longer feasible. The team uses deep neural networks for automated segmentation to isolate paint layers from backgrounds. They then apply Content-Based Image Retrieval (CBIR) using Gabor filters and color spatial distribution moments to quantify similarities between different artworks, helping to identify hidden connections between artists and historical periods.

2. Manifold Learning for Archaeological Pottery

Archaeologists often struggle with the inconsistent classification of Roman pottery shapes across decades of literature. To systematize this, the researchers utilize a dataset of over 5,000 pottery profiles. By employing sparse autoencoders, they project these complex shapes into a lower-dimensional latent subspace. This mathematical reduction allows for hierarchical clustering, creating a "tree of life" for pottery that remains robust even when dealing with fragmented or partial profiles common in excavations.

3. Variational Inpainting for Illuminated Manuscripts

Restoring fragile manuscripts requires non-invasive techniques. The project employs Partial Differential Equations (PDEs) for digital inpainting, repairing small scratches and larger holes by sampling from intact parts of the image. A particularly innovative approach uses drift-diffusion osmosis equations to fuse multi-spectral data (such as infrared) into visible-light images. This allows conservators to "see through" historical overpainting, such as when political or personal changes led to the alteration of original heraldry in prayer books.

Data-Driven Insights and the Human-in-the-Loop

A central insight of the MACH project is that mathematical objectivity must be balanced by expert validation. In art conservation, there is rarely a single "ground truth" as found in engineering; instead, success is defined by whether the results align with or challenge an expert's specialized knowledge.

Clustering visualization of painting cross-sections, enabling the identification of artistic patterns across large datasets.

The software toolkits developed by the team act as "indispensable mediators." They provide a common interface where art experts can interact with mathematical results, providing feedback that drives further algorithmic refinement. This iterative loop ensures that the tools remain intuitive and relevant to the humanities community.

Results and Future Directions

The project has successfully moved from theoretical mathematics to functional, user-friendly toolkits. Qualitatively, the manuscript toolkit has demonstrated the ability to virtually "undo" historical alterations, such as the overpainting of coat of arms in the prayer book of Duke Francis I of Brittany. Quantitatively, the pottery classification system provides a coherent framework that resolves long-standing inconsistencies in archaeological literature.

Future work involves the official public release of these toolkits and the development of new algorithms inspired by the unique challenges of the art world. As the project expands, the goal remains to ensure these tools are accessible to the global arts and humanities community, fostering a future where math and art speak the same language.

Summary updated

Abstract

Our interdisciplinary team Mathematics for Applications in Cultural Heritage (MACH) aims to use mathematical research for the benefit of the arts and humanities. Our ultimate goal is to create user-friendly software toolkits for artists, art conservators and archaeologists. In order for their underlying mathematical engines and functionality to be optimised for the needs of the end users, we pursue an iterative approach based on a continuous communication between the mathematicians and the cultural-heritage members of our team. Our paper illustrates how maths can speak art, but only if first art speaks maths.
BibTeX
@misc{leone2020art,
  author = {Ninetta Leone and Simone Parisotto and Katarzyna Targonska-Hadzibabic and Spike Bucklow and Alessandro Launaro and Suzanne Reynolds and Carola‐Bibiane Schönlieb},
  title = {{Art Speaks Maths, Maths Speaks Art.}},
  year = {2020},
  eprint = {2007.08886},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2007.08886},
}
Figure