TL;DR
Traditional visual deformation measurement relies on data-heavy high-speed cameras to track complex, rapid surface motion. This paper introduces an event-frame hybrid framework that pairs the high temporal resolution of event cameras with the spatial precision of standard frames. By linearizing deformations over triangular meshes and using a neighborhood-guided optimization strategy, the method achieves a 1.6× higher tracking survival rate under large displacements while slashing data storage and processing costs by over 80%.
What Problem Does This Paper Address?
Visual Deformation Measurement (VDM) is a non-contact technique used to recover dense deformation fields by tracking surface motion. It is vital for structural health monitoring, material mechanics analysis, and biomechanics research.
Unlike rigid body tracking—which only requires estimating 6 degrees of freedom—deformable surfaces have massive degrees of freedom because every surface point can move independently. Traditional image-based methods handle this by using expensive high-speed cameras to ensure minimal motion between consecutive frames, which keeps the correspondence search space manageable. However, this approach creates a massive bottleneck: processing and storing thousands of redundant frames incurs prohibitive computational and storage overhead, severely limiting practical scalability.
Methods
The authors propose an event-frame fusion framework that exploits the complementary nature of two modalities: event cameras capture rapid motion dynamics asynchronously without data redundancy, while conventional low-frame-rate images provide low-noise, spatially dense observations.
The core methodology consists of two main innovations:
- Affine Invariant Simplicial (AIS) Framework: The non-linear deformation field of an object's surface is systematically divided into multiple local triangular sub-regions (simplices). Within each triangle, the deformation parameters are linearized using an affine transformation. This reduces a high-dimensional problem into a low-parametric vertex tracking task. Events are then associated with these sub-regions using a "same-side test" and warped to a reference timestamp via barycentric interpolation to maximize event contrast.
- Neighborhood-Greedy Optimization Strategy: Long-term dense tracking often suffers from local error accumulation, which can cause global tracking failure. The authors introduce a strategy that evaluates the convergence quality of each sub-region. Well-converged regions are fixed and act as anchors, leveraging physical strain continuity priors to guide and correct their poorly-converged neighbors.
The optimization follows a hierarchical coarse-to-fine paradigm, solving for global rigid self-motion first before iteratively splitting and refining the mesh for fine-grained deformations.
Datasets
Because no standard event-based datasets existed for VDM tasks, the authors built a custom hardware setup consisting of a Prophesee EVK4 event camera and a 210 fps grayscale camera collocated behind a 50:50 beam splitter.
Using this system, they introduced a new benchmark dataset featuring:
- Over 120 real-world sequences spanning diverse deformation modes (squeezing, stretching, bending, and cracking).
- Displacements ranging from small-scale (<20 pixels) to large-scale (100+ pixels).
- Highly accurate ground-truth deformation fields derived by running state-of-the-art VDM algorithms directly on the high-speed 210 fps video recordings.
Key Insights and Ideas
- Local Linearization of Complex Fields: Complex, non-linear global deformations can be effectively modeled by a collection of locally linear, affine-invariant triangular patches. This low-parameter representation successfully mitigates the motion ambiguities typically caused by sparse and noisy event data.
- Greedy Collaboration Prevents Drift: Instead of optimizing the entire high-dimensional field simultaneously—which is slow and prone to local minima—allowing successful regions to "pull" failing neighbors along via continuity constraints prevents tracking drift. This neighborhood-greedy approach yields a threefold speedup in mean convergence time (from 26.5s down to 7.2s).
- Efficiency Without Sacrifice: High-frequency deformation dynamics do not strictly require high-speed video. Low-frame-rate frames (e.g., 5 fps) combined with asynchronous event streams retain enough spatiotemporal information to match or exceed high-speed video tracking accuracy.
Key Results
The framework was evaluated against several baselines, including OpenCorr (model-based VDM), StrainNet (learning-based VDM), E-RAFT (event-based optical flow), and CoTrackerV3 (long-term point tracking).
- Tracking Robustness: In scenarios with large continuous displacements (100+ pixels), the proposed method achieved a 65.7% survival rate (the fraction of sequence duration before tracking fails), outperforming the top baseline, CoTrackerV3, by 1.6×.
- Resource Efficiency: Remarkably, when utilizing a 5 fps frame input alongside the event stream, the method achieved tracking accuracy comparable to traditional methods running on 100 fps video while consuming only 18.9% of the data storage and processing resources.
Limitations and Future Work
The authors note a few key limitations of the current system:
- Topological Changes: Because the framework relies heavily on spatial continuity assumptions, it can generate "pseudo-strains" if an object undergoes a topological break, such as a material cracking completely apart.
- Specularities: Performance degrades in scenes with strong specular reflections, which violate the brightness constancy assumption.
Future Directions: The authors plan to extend the framework to 3D deformation fields using synchronized event camera arrays for volumetric strain analysis. They also aim to explore purely event-driven VDM systems to further minimize data throughput and deploy the framework into real-time closed-loop applications like robotic manipulation of deformable objects.
Summary updated