Deep Learning and Synthetic Media

Raphaël MillièrearXiv 2022 · DOI · arXiv

Abstract

Deep learning-based synthetic audiovisual media (DLSAM), commonly referred to as "deepfakes," represents more than a mere evolution of computer-generated imagery (CGI). This paper argues that DLSAM constitutes a fundamental ontological shift in media production. By transitioning from discrete, manual manipulation to the navigation of continuous, high-dimensional latent spaces, deep learning blurs the traditional boundaries between archival recordings of reality and purely synthetic artifacts.

Introduction: Beyond the Ethics of Deepfakes

While public discourse on deepfakes has centered on ethical concerns such as disinformation and non-consensual imagery, there exists a significant gap in our ontological understanding of the technology. Traditional media taxonomies are ill-equipped to categorize modern DLSAM. For instance, is a smartphone photograph enhanced by AI "super-resolution" an archival record or a synthetic generation? Does a deepfake represent a more efficient version of traditional visual effects (VFX), or is it qualitatively different?

This research addresses these questions by proposing a rigorous framework for classifying media and explaining how the underlying mechanics of deep learning—specifically manifold learning and disentanglement—represent a "discontinuity" from previous synthesis techniques.

A New Taxonomy of Audiovisual Media

The core contribution of this work is a novel taxonomy based on the etiology (origin) of media. This framework moves beyond simple "real vs. fake" binaries to consider how media is constructed across several dimensions:

  1. Hand-made vs. Machine-made: Distinguishing between human-crafted art (like a painting) and media produced via mechanical or digital processes.
  2. Archival vs. Synthetic: Archival media (e.g., raw photography) serves as a "transparent" record of reality, whereas synthetic media involves a generative component.
  3. Global vs. Local Manipulation: Differentiating between broad applications (like a color filter) and specific, localized changes (like face swapping).
Figure 1: A comprehensive taxonomy of audiovisual media, mapping traditional and deep learning-based techniques.

This taxonomy allows researchers to categorize a wide spectrum of media, from 19th-century symphonies to modern text-to-video generation, highlighting where deep learning introduces new categories of "partially synthetic" content.

The Mechanics of Deep Learning Synthesis

The qualitative shift in DLSAM is driven by the Manifold Hypothesis. This hypothesis suggests that high-dimensional data (such as the set of all possible 512x512 images) actually resides on a lower-dimensional, continuous "surface" or manifold. Deep learning models, such as Generative Adversarial Networks (GANs) and Variational Autoencoders (VAEs), learn to navigate this manifold.

From Discrete Editing to Latent Steering

Traditional editing is ad hoc and discrete—cutting and pasting specific pixels or frames. In contrast, DLSAM is sampled. Creating a deepfake involves finding a specific path through a mathematical latent space where every point represents a plausible, high-fidelity image or sound.

Figure 3: The architecture of a Generative Adversarial Network (GAN), showing the interplay between a generator and a discriminator.

Disentanglement as Control

A key innovation in deep learning is the emergence of disentangled representations. In a well-trained model, specific attributes like "age," "lighting," or "pose" become independent vectors within the latent space. This allows for "steering" media in real-time by adjusting these "sliders," a feat that would require immense manual labor in traditional VFX pipelines.

The Archival-Synthetic Continuum

The research argues that the boundary between "real" and "fake" is increasingly a gradient rather than a hard line. Modern mobile photography frequently employs deep learning for "night mode" or "super-resolution," effectively synthesizing visual details that were never captured by the physical sensor. Consequently, even "raw" digital photos are becoming partially synthetic.

This leads to the concept of Navigable Media: a new medium situated between cinema and video games. We are entering an era of photorealistic video that can be steered or interacted with in real-time by manipulating latent variables, challenging the historical "transparency" of the photographic record.

Conclusion and Future Outlook

DLSAM represents a qualitative shift in media history for three primary reasons:

  1. Democratization: Professional-grade synthesis no longer requires a specialized VFX studio, only a GPU and a relevant dataset.
  2. Novel Capabilities: Tasks such as real-time voice cloning or inverting a 2D photo into a manipulatable 3D model are fundamentally new.
  3. Ontological Blurring: The mechanical connection to reality is increasingly mediated by generative models.

While challenges remain—particularly regarding spatiotemporal consistency in long-form video—the future of DLSAM lies in its potential as a creative paradigm. Moving beyond the negative connotations of the "deepfake" label, this technology offers a new way to assign truth value and creative agency in an era where the archival and synthetic are inextricably linked.

Summary updated

Abstract

Deep learning algorithms are rapidly changing the way in which audiovisual media can be produced. Synthetic audiovisual media generated with deep learning - often subsumed colloquially under the label "deepfakes" - have a number of impressive characteristics; they are increasingly trivial to produce, and can be indistinguishable from real sounds and images recorded with a sensor. Much attention has been dedicated to ethical concerns raised by this technological development. Here, I focus instead on a set of issues related to the notion of synthetic audiovisual media, its place within a broader taxonomy of audiovisual media, and how deep learning techniques differ from more traditional approaches to media synthesis. After reviewing important etiological features of deep learning pipelines for media manipulation and generation, I argue that "deepfakes" and related synthetic media produced with such pipelines do not merely offer incremental improvements over previous methods, but challenge traditional taxonomical distinctions, and pave the way for genuinely novel kinds of audiovisual media.
BibTeX
@misc{milliere2022deep,
  author = {Raphaël Millière},
  title = {{Deep Learning and Synthetic Media}},
  year = {2022},
  eprint = {2205.05764},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2205.05764},
}
Figure