Abstract
Reconstructing and tracking deformable surfaces with little or no texture has posed long-standing challenges. Fundamentally, the challenges stem from textureless surfaces lacking features for establishing cross-image correspondences. In this work, we present a novel type of markers to proactively enrich the object's surface features, and thereby ease the 3D surface reconstruction and correspondence tracking. Our markers are made of fluorescent dyes, visible only under the ultraviolet (UV) light and invisible under regular lighting conditions. Leveraging the markers, we design a multi-camera system that captures surface deformation under the UV light and the visible light in a time-multiplexing fashion. Under the UV light, markers on the object emerge to enrich its surface texture, allowing high-quality 3D shape reconstruction and tracking. Under the visible light, markers become invisible, allowing us to capture the object's original untouched appearance. We perform experiments on various challenging scenes, including hand gestures, facial expressions, waving cloth, and hand-object interaction. In all these cases, we demonstrate that our system is able to produce robust, high-quality 3D reconstruction and tracking.
The Challenge of Textureless Surfaces
Tracking the deformation of textureless objects—such as a blank sheet of paper or a white silk scarf—is a fundamental "chicken-and-egg" problem in computer vision. To train robust tracking models, researchers need high-quality ground-truth data. However, obtaining this data for featureless surfaces is nearly impossible because traditional algorithms rely on surface textures like patterns or edges to establish correspondences. If physical markers are added to the object to facilitate tracking, the surface is no longer textureless, and the model risks learning to track the markers themselves rather than the underlying material properties.
Methodology: Invisible Markers and Time-Multiplexed Imaging
To break this cycle, we developed a hardware-software co-design approach that utilizes "invisible" markers. These markers are made of fluorescent dyes that remain invisible under normal light but glow vibrantly (in red or blue) under ultraviolet (UV) light.
Our system employs a specialized "rhombicuboctahedral" camera rig consisting of 42 cameras and 60 UV LED units. By interleaving frames at 60 fps—one with UV lights on and the next with them off—we capture synchronized pairs of images. The UV frames provide the high-contrast features necessary for 3D reconstruction and correspondence tracking, while the visible-light frames capture the object's natural, textureless appearance. A template-fitting algorithm using an embedded deformation graph aligns a 3D model to the marker-derived point cloud, which is then projected back onto the marker-free frames to provide ground-truth labels.
The DOT Dataset
Using this system, we curated the DOT (Deformable Object Tracking) dataset. It comprises approximately one million video frames across 200 motion sequences, covering four main categories: 1D ropes, 2D stiff paper, 2D highly deformable cloth, and complex articulated hands. This dataset provides a unique resource for training models on surfaces ranging from repetitive patterns to "pure white" materials with zero visible features.
Key Insights and Results
A critical advantage of our approach is the ability to decouple appearance from tracking. By using different fluorescent colors (e.g., UV-blue for hands and UV-red for objects), the system can robustly track complex interactions without feature confusion.
We benchmarked state-of-the-art models like RAFT and PWC-Net on the DOT dataset. Initial results showed that existing models failed significantly on "pure white" sequences—a phenomenon we call the "Textureless Wall." However, fine-tuning these models on DOT led to dramatic improvements, reducing errors to sub-pixel accuracy. Furthermore, models trained on DOT demonstrated superior cross-dataset generalization, suggesting they learn the fundamental physics of non-rigid deformation rather than just dataset-specific patterns.
Conclusion and Future Work
While the current system requires specialized hardware and manual marker application, it provides a path forward for creating large-scale, accurately labeled datasets for challenging materials. Future work will explore high-resolution UV printing for denser marker patterns and the development of foundation models for general non-rigid motion.
Summary updated