While deep-fake creators have mastered the art of synthesizing realistic faces and mouths, they have largely overlooked the ears. This research demonstrates that the human ear provides a powerful forensic signal: not only is its static shape a unique biometric "fingerprint," but its subtle movements are physically coupled with the jaw and mouth—a relationship that current deep-fake generators fail to replicate.
The Aural-Oral Connection
The "cat-and-mouse" game of deep-fake detection usually focuses on the face, looking for eye-blinking inconsistencies, mouth artifacts, or blending glitches. However, as synthesis techniques like Generative Adversarial Networks (GANs) improve, these low-level artifacts are disappearing.
The authors identify a significant "blind spot" in deep-fake generation: the ear. Face-swaps replace the face but leave the original person's ears, creating a biometric mismatch. Lip-sync fakes modify the mouth to match new audio but ignore the fact that moving the mandible (lower jaw) physically pulls on the ear canal and outer ear. By ignoring these "aural and oral dynamics," deep-fakes create a detectable decoupling between how a person speaks and how their ears move.
Forensic Methodology
The researchers proposed a two-pronged forensic approach to exploit these weaknesses.
1. Aural Dynamics (The "Sync" Test)
To detect if ear movement matches speech, the authors developed a pipeline to measure correlation. They manually annotate 35 landmarks on the ear, including the helix, lobule, and tragus, and track them using a Kanade-Lucas-Tomasi (KLT) tracker.
They use Bezier curves to model the ear's shape and estimate local motion via dense optical flow. This ear motion is then compared against two "oral" signals: the vertical distance between lips (mouth openness) and the Audio RMSE (loudness of the speech). A logistic regression model is trained on these correlations to distinguish between authentic synchronized movement and the "broken" dynamics of a deep-fake.
2. Aural Biometrics (The "Identity" Test)
To detect face-swaps, they treat the ear as a static biometric. They align the ear from a video to a known "reference ear" of the target individual using a planar homography. They then measure the average Euclidean distance between the shapes to see if the ears actually belong to the person being impersonated.
Key Results and Insights
The study utilized a custom dataset of 64 high-quality YouTube videos featuring public figures like Joe Biden, Angela Merkel, Donald Trump, and Mark Zuckerberg. The dynamic model achieved an average AUC (Area Under the Curve) of 0.84 when trained across all individuals. When the model was trained on a specific individual, the accuracy reached an AUC of up to 0.98 for GAN-generated fakes.
A notable case study involved the viral "@deeptomcruise" TikTok videos. While visually stunning, biometric analysis showed that the impersonator’s ear shape differed from the real Tom Cruise’s by over 35% more than the natural variation between different photos of the real Cruise.
Discussion and Limitations
Unlike low-level pixel analysis, this method relies on high-level semantic features (movement and shape), making it more resilient to common "laundering" techniques like resizing or video compression. However, several limitations remain:
- Occlusion: Hair covering the ears prevents the use of this method.
- Head Movement: Large or rapid head rotations can cause tracking slippage.
- Reference Data: The biometric approach requires a high-quality reference image of the target's ear.
Future work aims to automate the initial ear landmarking process and expand the "oral" signals to include other facial movements like eyebrow raises or smiles, which also induce subtle ear motions.
Summary updated