The reconstruction of fragmented historical textiles—like those found in the 9th-century Oseberg ship burial—is one of the most grueling tasks in archaeology. While deep learning has mastered solving digital jigsaw puzzles of natural images, it has historically stumbled when faced with the faded, degraded, and repetitive patterns of ancient weaving.
A new paper from researchers at the Norwegian University of Science and Technology (NTNU) introduces TEXMET, the first large-scale benchmark specifically designed to bridge the gap between "clean" computer vision and the messy reality of cultural heritage.
TL;DR
The authors present TEXMET, a curated dataset of over 18,000 historical textile images, and demonstrate that standard AI puzzle-solvers fail on archaeological data without domain-specific training. By fine-tuning a graph-based diffusion model on TEXMET, they boosted reconstruction accuracy on real-world Viking Age fragments from a meager 26.67% to 64.44%.
What Problem Does This Paper Address?
Standard "jigsaw puzzle" benchmarks in AI typically use datasets like ImageNet, where models learn to reassemble clean, high-contrast images of dogs, cars, or landscapes. Archaeological textiles, however, present a unique set of "nightmare" features:
- Severe Degradation: Centuries of burial lead to fiber loss, staining, and fading.
- Repetitive Patterns: Unlike a landscape with a clear horizon, a textile might consist of thousands of nearly identical weave intersections.
- Irregular Shapes: Real fragments don't come in neat square tiles.
The authors identified a massive "domain gap": models trained on natural images simply don't know what to look for when trying to piece together a 1,200-year-old tapestry.
Methods
The researchers evaluated three state-of-the-art (SOTA) puzzle-solving architectures to see which could handle the transition to textiles:
- JPDVT (Diffusion Vision Transformer): Treats reassembly as a denoising problem, iteratively moving pieces to their correct spots.
- FCViT (Coordinate Regression): Attempts to predict the exact X-Y coordinates for every fragment simultaneously.
- DiffAssemble (Graph-Based Diffusion): Represents the puzzle as a graph where fragments are nodes and spatial relationships are edges. This model is unique because it can handle rotations and varying puzzle sizes.
The core of their approach was domain adaptation: taking these models (initially trained on natural images) and fine-tuning them on their new textile dataset to see if they could learn the "language" of ancient weaving.
Datasets
- TEXMET (New): The star of the paper. It contains 18,644 high-quality historical textile images harvested from the Metropolitan Museum of Art’s Open Access collection. The authors used CLIP embeddings and UMAP clustering to filter out "noise" (like images of armor or weapons misclassified as textiles) to ensure a clean, diverse training set.
- Oseberg Fragments: A set of 45 real, highly degraded fragments from the Oseberg ship burial (834 CE). These fragments are extremely challenging, with an average "masked area" (missing/degraded parts) of over 75%.
- ImageNet-1k: Used as the baseline for "natural image" pre-training.
Key Insights and Ideas
- The Coordinate Collapse: FCViT, which relies on regressing coordinates, experienced a near-total collapse when moved to textiles (dropping to 8.3% accuracy). This suggests that coordinate-based models are "brittle" and overly dependent on the specific visual features of their training data.
- Graphs are Greater than Grids: DiffAssemble’s graph-based approach proved the most robust. By focusing on the relationships between pieces rather than just their absolute positions, it maintained much higher performance across different domains.
- Learning Patterns, Not Just Edges: To prove the models weren't just "cheating" by matching jagged edges, the authors performed "inscribed cropping" (removing the edges of the pieces). The models still performed well, proving they had actually learned to recognize complex textile weave patterns.
- Fine-Tuning is Non-Negotiable: The "aha moment" was the dramatic jump in performance on the Oseberg fragments. Pre-trained models were essentially guessing, but after seeing the MET textile data, they became significantly more "literate" in archaeological textures.
Key Results
The results highlight a massive improvement through domain adaptation:
- TEXMET Accuracy: The fine-tuned DiffAssemble model achieved 85.89% puzzle accuracy on the textile benchmark.
- The Oseberg Leap: On the real-world, highly degraded Oseberg fragments, the model's accuracy jumped from 26.67% to 64.44% after fine-tuning on TEXMET.
- Robustness: DiffAssemble proved to be the most resilient to "archaeological noise" (irregular masking and fading) compared to the transformer-only models.
Limitations and Future Work
While a major step forward, the authors acknowledge several hurdles:
- The "Square" Constraint: Most current models still require fragments to be divided into regular square grids. Real archaeological fragments are jagged and irregular.
- Scale: The Oseberg evaluation was limited to 45 fragments. Scaling this to the hundreds or thousands of fragments found in a typical excavation remains a challenge.
- Cross-Fragment Matching: This study focused on reassembling pieces within a single known fragment. The ultimate goal—matching hundreds of different fragments to reconstruct an entire lost tapestry—is the next frontier.
The researchers have released the TEXMET dataset and their code publicly, providing a new foundation for the computational archaeology community to build upon.
Summary updated