Facial Performance Capture and Synthesis Under Heavy Occlusion

This direction is split into other directions: facial occlusion in head-mounted displays. Its last version stays readable and citable here.

Version 1 of 144 papers54 conceptsfrozen 2026-10-04

When we interact face-to-face, we rely on a constant stream of non-verbal cues: the crinkle of eyes during a laugh, the furrow of a brow in confusion, or the subtle direction of a gaze. However, modern wearable technology—like Virtual Reality (VR) headsets and Augmented Reality (AR) glasses—acts as a "social blindfold," physically obstructing the very features we use to communicate.

Researchers are developing a suite of techniques to "see through" these obstructions, reconstructing full facial expressions and emotional states from partial, distorted, or even non-visual data.

The Perspective Challenge: Up-the-Nose and Oblique Views

The primary hurdle in wearable facial capture is not just that the face is covered, but where the cameras must be placed. To keep headsets compact, cameras are often mounted on the rim, looking at the face from extreme, "foreshortened" angles.

Mohamed Elgharib and colleagues explored this in EgoFace, where a single fisheye camera mounted on eyeglass frames captures a highly distorted "up-the-nose" view. The intuition behind solving this is domain translation: rather than trying to "un-distort" the image geometrically, the system uses a Generative Adversarial Network (GAN) to map the features it can see (like the jawline and mouth) onto a pre-learned model of a natural frontal face.

Similarly, Chaitanya Patel and Shih-En Wei address the "domain gap" in VR headsets. Cameras inside a headset see the face in infrared (IR) from inches away, which looks nothing like a person in natural light. Their work uses iterative transformers to bridge this gap, refining noisy IR signals into high-fidelity avatar animations in seconds.

Inferring the Whole from the Periocular Part

If the top half of the face is hidden by a display, can we still know what the user is feeling? Research suggests the answer lies in the periocular region—the area immediately surrounding the eyes.

In Eyemotion, Steven Hickson and Nick Dufour demonstrated that internal eye-tracking cameras capture enough muscle movement (Action Units) to infer full-face expressions. While a human might struggle to recognize an emotion from just an IR close-up of an eye, deep learning models can identify the subtle skin stretches and folds that correlate with a smile or a frown hidden elsewhere.

For environments with strict power constraints, Bo Dong and Felix Heide introduced SEEN, which uses bio-inspired event cameras. These sensors only record changes in light (movement), allowing the system to track rapid eye blinks and micro-expressions with minimal energy, even when traditional cameras would be blinded by the dark interior of a headset.

Digital De-occlusion and Reverse Pass-Through

Once the system understands the hidden expression, the next step is to "remove" the headset for the observer. This is known as facial de-occlusion.

Yajie Zhao and Ruigang Yang pioneered this with Mask-off, a framework that synthesizes a photo-realistic, unoccluded face in real-time. By combining a 3D head model with live IR data from inside the headset, they "paint" the user's actual eyes back onto a video feed, restoring eye contact.

A more recent evolution of this is Reverse Pass-Through. In Eye-See-You, Ankan Dash and Guiling Wang describe a system that reconstructs the user's eyes for an external display on the headset itself. This allows people in the physical room to "see" the user's eyes, breaking the isolation of VR.

Beyond Vision: Decoding the Mind

When occlusion is total or cameras are impractical, researchers look to physiological signals. Haolin Xiong and Yajie Zhao recently pushed this to the limit with Mind-to-Face. Instead of cameras, this system uses EEG sensors to decode brain activity directly into 3D facial expressions.

This "neural-driven" approach bypasses the physical face entirely. If the brain intends to smile, the avatar smiles, regardless of whether a headset, a medical mask, or life-support equipment (Xi Cheng et al., SeeMe) is blocking the view.

Why It Works: The Power of Priors

The common thread across these papers is the use of personalized priors. Because a user's face structure doesn't change, systems like those proposed by Surabhi Gupta and Ashwath Shetty in their occlusion removal work only need a few minutes of unoccluded video to learn a person's specific expressions.

By anchoring the noisy, partial data from wearable sensors to a high-quality digital double, these models don't have to "guess" what a face looks like—they only have to figure out how the existing face is currently moving. This shift from general image synthesis to person-specific refinement is what allows these systems to run in real-time on mobile hardware, finally bringing natural social presence to occluded environments.

Papers in this version

History & citationVersion 1 · 2026-10-04

Split into other directions · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_facial_performance_capture_and_synthesis_8267ce04,
  title = {Facial Performance Capture and Synthesis Under Heavy Occlusion},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/facial-performance-capture-and-synthesis-under-heavy-occlusion/rdr_8267ce0412b6438cbb57a052a2ce6a1f},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_facial_performance_capture_and_synthesis_8267ce04_v1,
  title = {Facial Performance Capture and Synthesis Under Heavy Occlusion},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/facial-performance-capture-and-synthesis-under-heavy-occlusion/rdr_8267ce0412b6438cbb57a052a2ce6a1f/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}