TL;DR
TimeWalker reconstructs and animates realistic 3D head avatars across a person’s entire lifetime using only unstructured, in-the-wild photo and video collections. By pairing an additive neural basis-blending module with a dynamic 2D Gaussian splatting representation, it disentangles age/life stage, facial expression, shape, and viewpoint—enabling steerable "time travel" and reenactment for historical and contemporary figures.
What Problem Does This Paper Address?
Current 3D head avatar techniques implicitly assume an individual's facial identity and shape are invariant, capturing subjects at a single moment in time (e.g., from synchronized multi-camera rigs or short video clips). In reality, human identity evolves across decades: facial bone and tissue structures shift, wrinkles develop, and hairstyles and accessories change.
Modeling a personalized 3D avatar on a lifelong scale introduces three central hurdles:
- Long-horizon identity drift: Drastic physical changes over time make it difficult to preserve the target subject's core identity.
- Scattered, uneven data: We cannot capture a lifetime in a multi-view laboratory rig; available lifelong data consists of unstructured, uneven, and often single-view photos/videos across disparate lighting and poses.
- Full-scale animation control: Avatars need to be independently controllable in expression, head pose, camera view, and life stage without entangled artifacts.
Methods
TimeWalker handles these challenges via a personalized neural parametric formulation built on top of the FLAME face model:
[Canonical Mean Head] + ∑ w_i [Neural Head Basis_i] ──> [Moment-Specific Head] ──(FLAME Motion Warping)──> [Animated Avatar]
- Neural Linear Combination Space: Analogous to classical 3D Morphable Models (3DMMs), TimeWalker represents any life stage as an additive combination of a canonical mean head (shared across all ages) and moment-specific attribute deformations.
- Dynamo (Dynamic Neural Basis-Blending Module): Instead of assigning fixed representations to each age, Dynamo learns a compact set of multi-resolution hashgrid feature bases. It dynamically monitors blend weights $w_i$; bases consistently falling below a threshold $\kappa$ are pruned during training. This forces the network to learn reusable, identity-defining characteristics rather than memorizing superficial appearances.
- DNA-2DGS (Dynamic 2D Gaussian Splatting): Standard 3D Gaussians cause depth ambiguity along discontinuous surfaces. TimeWalker adopts 2D Gaussian Surfels (oriented planar disks) anchored on the canonical FLAME mesh. To animate expressions, it computes a deformation gradient tensor $F$ from tracked FLAME mesh triangles to warp surfels into dynamic motion space.
- Defer-Warping for Meshing: Running Poisson surface reconstruction on dynamic point sequences is prohibitively slow and prone to motion blur. TimeWalker runs Poisson reconstruction once on the static moment-specific surfels to obtain a clean surface mesh, and then applies motion deformations directly to the resulting mesh vertices.
Datasets
- TimeWalker-1.0 (New Benchmark): The authors curate a lifelong dataset consisting of 40 celebrities gathered from high-resolution YouTube videos and films. It contains over 2.7 million frames (15k–260k frames per identity), covering ages 20–80 across diverse genders, ethnicities, head poses, and illumination conditions.
- Evaluation Datasets: The authors also evaluate and test cross-identity reenactment using existing benchmarks: INSTA, Nersemble, and RenderMe-360.
Key Insights and Ideas
- Shared Canonical Base as an Invariant Anchor: By running an initial warm-up phase that optimizes surfels to fit all life stages simultaneously, the network learns an individual's "average" lifetime face. This prevents overfitting to rare camera angles or uneven data within sparse life stages.
- Pruning Redundant Bases: Rather than 1:1 mapping between life stages and neural bases, Dynamo dynamically disables uninformative bases. This compact formulation reduces memory while improving rendering fidelity over fixed-basis models.
- Decoupled Animation Pipeline: By separating moment-specific deformation (learned via Dynamo) from motion-specific warping (driven by FLAME expression coefficients), facial expressions can be retargeted across different decades of a person's life without altering their age-specific identity.
Key Results
- Lifelong Modeling (1 Model vs. All Life Stages): On TimeWalker-1.0, TimeWalker achieves 27.28 PSNR / 0.949 SSIM / 0.071 LPIPS, substantially outperforming baselines adapted for multi-stage inputs like INSTA++ (26.39 / 0.879 / 0.139) and FlashAvatar (22.14 / 0.771 / 0.267).
- Parity with Multi-Model Baselines: Even when competing against separate models trained independently for each individual life stage (1-vs-N), a single TimeWalker model achieves superior perceptual quality (LPIPS of 0.071 vs. Gaussian Surfels++ at 0.134).
- Downstream Applications: The learned representation supports cross-lifestage and cross-identity reenactment, as well as text-guided 3D editing (via diffusion models like DGE) while preserving geometry and decoupling facial expressions.
Limitations and Future Work
- Expression Extremes and Mouth Interiors: Because the motion warping relies on FLAME topology, the avatar struggles with extreme or exaggerated facial expressions and unmodeled internal mouth structures (such as the tongue).
- Age Range Coverage: The current framework focuses on adult life stages (ages 20–80); modeling human growth from infancy through adolescence remains unaddressed due to radical anatomical changes and data scarcity.
- Future Directions: Incorporating pre-trained 3D generative diffusion priors could help fill in unobserved views and improve rendering of challenging facial interactions.
Summary updated