Depth-Localized Latent Directions: Learning Where in the Generator Each Edit Lives

Image generators such as StyleGAN contain usable editing controls: directions in their internal code that rotate a face, age it, or relight it. Several methods find these directions without labels. In all of them, a human still decides which layers of the generator a direction should be applied to. Coarse layers control pose and shape, and fine layers control color and texture, but the boundary between them is chosen by trial and error. We propose to learn each direction together with a soft mask over generator depth, so that every discovered edit also states which band of layers it acts on. The objective rewards edits that are distinguishable from one another and confined to as few contiguous layers as possible.

The second claim matters more than the editing tool. We hypothesize that how tightly an edit is confined in depth can be measured without labels and predicts how disentangled it is. If that holds, it gives a partial answer to the model-selection problem raised by Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representations: the generator's layer hierarchy supplies the structural assumption, and a depth-compactness score supplies a way to rank directions and training runs without ground truth.

Core Claim

Hierarchical feature learning shows that deep networks build representations in stages, from local texture to part to global layout. Removing a stage hurts, as ImageNet Classification with Deep Convolutional Neural Networks found when it ablated convolutional layers. Return of the Devil in the Details showed that these learned stages outperform shallow multiscale pipelines even when data augmentation is equalized.

A generator should inherit the same staging in reverse. If so, a factor that is genuinely independent, such as azimuth or hue, should be controllable from a narrow band of layers. An entangled bundle, such as pose plus identity plus lighting, should need many layers spread across depth.

We claim two things, each testable on its own. First, jointly learning direction and depth mask produces edits that are purer than those from fixed or manually chosen layer ranges. Second, the entropy of a direction's per-layer effect profile, computed from the frozen generator alone, correlates with ground-truth disentanglement scores across directions and across training seeds.

Locatello's Theorem 1 is not contradicted here. The theorem rules out disentanglement without assumptions about structure. Our assumption is explicit: factors align with contiguous depth bands of a hierarchical generator. The experiments test whether that assumption holds.

Relation to Prior Work

Three methods already use generator depth, and all three fix the layer choice by hand.

  • GANSpace runs principal component analysis on StyleGAN's intermediate codes. It shows that applying a component to all layers entangles attributes, and that restricting it to a manually chosen layer subset cleans the edit up. This is the motivating evidence for our work.
  • Closed-Form Factorization of Latent Semantics in GANs, known as SeFa, eigendecomposes the generator's concatenated projection weights. The user still picks which layers to concatenate.
  • LatentCLR trains direction models contrastively on the feature changes at one user-chosen layer.

Our method differs from all three in that the layer assignment is a learned output of discovery rather than an input to it.

Two other methods supply the mechanism we reuse.

  • We take the label-free training signal from Unsupervised Discovery of Interpretable Directions in the GAN Latent Space. Its "reconstructor game" trains a small network that sees an original and an edited image and must guess which direction was applied and by how much.
  • We adopt orthogonality through Householder reflections from Householder Projector. That paper applies the projector per layer without coordinating across depth, and supplies the Perceptual Interpretable Path Length (PIPL) metric we use for evaluation.

The hierarchical generative models occupy a different position. FineGAN hardcodes background, shape, and appearance stages. Hierarchical Diffusion Autoencoders (HDAE) builds a multi-resolution latent but derives its edits from supervised classifiers. Both impose the hierarchy rather than discover how factors distribute across it.

DisDiff exploits the coarse-to-fine order of diffusion timesteps rather than architectural depth. It is our strongest quantitative comparison on synthetic factors.

Research Program

We work with StyleGAN-family generators, which receive a separate style vector at each of $L$ layers. We learn $K$ directions $v_k$, kept orthonormal by the Householder parameterization. Each direction also gets a gate vector $g_k \in [0,1]^L$ that says how strongly to apply it at each layer. An edit of strength $\epsilon$ along direction $k$ sets the style input at layer $l$ to $w + \epsilon\, g_{k,l}\, v_k$.

The reconstructor $R$ sees the original image and the edited image and predicts $k$ and $\epsilon$. Directions, gates, and $R$ are trained jointly with the objective

$$\mathcal{L} = \mathcal{L}_{\text{cls}}(k) + \lambda_{\epsilon}\,\mathcal{L}_{\text{reg}}(\epsilon) + \lambda_{d}\sum_k \Big(H(\bar g_k) + \sum_{l}|g_{k,l}-g_{k,l+1}|\Big),$$

where the terms are defined as follows:

  • $\mathcal{L}_{\text{cls}}$ is the reconstructor's cross-entropy loss for identifying the direction.
  • $\mathcal{L}_{\text{reg}}$ is its regression loss for the edit strength.
  • $\bar g_k$ is the gate vector normalized to sum to one, and $H$ is its entropy. This term pushes each edit onto few layers.
  • The absolute-difference sum penalizes gaps between active layers, so the active layers form a contiguous band.
  • $\lambda_{\epsilon}$ and $\lambda_d$ weight the strength-prediction and depth terms.

A minimum total gate mass is enforced so that gates cannot collapse to zero.

The analysis object is the per-layer effect profile. For each direction we apply it at one layer at a time and measure the resulting perceptual change in the image. We then normalize these changes across layers and take the entropy, which we call depth entropy. This score uses only the frozen generator and requires no labels.

Baselines

The naive reference point is a set of random orthonormal directions applied to all layers. Next to it we run GANSpace and SeFa applied to all layers. These show what ignoring depth costs.

The manual-hierarchy baselines are GANSpace with the layer ranges its authors report and LatentCLR targeted at its published layer choice. All methods discover the same number of directions $K$ on the same generator checkpoints. All learned methods receive the same number of sampled latent codes and the same hyperparameter search budget. Each is run with five seeds.

For subjective quality, the file reports two user studies:

  • SeFa's directions were preferred over GANSpace's as more meaningful, 59% to 41% (Closed-Form Factorization).
  • 82% of LatentCLR's directions were judged meaningful (LatentCLR).

We will rerun a matched user study rather than compare against these numbers across different setups.

The strongest existing method on ground-truth factors is DisDiff. It reports a DCI disentanglement score of 0.337 on MPI3D, against 0.292 for the DisCo baseline (DisDiff). DCI measures how cleanly each true factor is captured by a separate learned code. Beating 0.337 under the same evaluation protocol would count as a clear win. Matching it with a GAN that is fed forward once rather than run through many denoising steps would still be useful.

The key ablation removes the gates' learning. We take the best directions from SeFa or GANSpace and give each a depth mask chosen post hoc, by keeping the contiguous layer band that holds most of its effect profile. If this two-stage pipeline matches joint learning, the editing contribution is not new.

Why not just…?

Why not just mask existing directions post hoc? This is the real threat. Gap 1 in the literature may be closable with a few lines of analysis on SeFa's output. Our expectation is that it will not fully close it. SeFa's eigenvalue imbalance, documented by Householder Projector, means its top directions already merge concepts across depth. No mask can split one merged vector into two clean edits, whereas joint learning can allocate separate directions to separate bands.

The ablation above settles the question directly. If post-hoc masking ties, we report that honestly and the paper rests on the second claim, depth entropy as a selection signal, which post-hoc masking does not address.

Why not just run SeFa separately on each block of layers? This needs a block partition, which is the manual step we aim to remove. It also cannot represent a factor whose band straddles a block boundary. We include per-block SeFa with a coarse/middle/fine split as a baseline. If its purity matches ours on faces, we test whether it degrades on generators where the manual split is less obvious, such as cars and full-body humans.

Why not just use CLIP text prompts or a few labeled pairs? This reintroduces the supervision bottleneck we aim to escape: the vocabulary is bounded by what text describes. Zero-Shot Rankability reports that text-derived axes for attributes ordered along a scale are suboptimal relative to axes fit with labeled pairs. Fine-Grained Activation Steering finds that coarse interventions interfere unless they are localized. The first finding tells against text-only steering; the second motivates our depth localization.

Why not just train a disentangled VAE? Locatello et al. found that the choice of objective explains about 37% of the variance in disentanglement scores, with seeds and hyperparameters dominating. DisDiff reports an FID of about 99.8 for disentangled VAEs on CelebA, versus 18.249 for its own model. FID is a standard measure of image realism, where lower is better. We include β-TCVAE on the synthetic benchmarks only, as a floor.

Proposed Experiments

Synthetic benchmarks with ground truth. We train StyleGAN2 on Shapes3D, Cars3D, and MPI3D, with 30 runs per dataset across seeds and $\lambda_d$ values. We follow the DisCo-style protocol: an encoder is trained to predict direction coordinates from generated images, and DCI is then computed on real images.

The decisive check for the second claim is a rank correlation between mean depth entropy and DCI, computed at two levels: across directions within a run, and across runs. A Spearman correlation reliably below −0.4 across all three datasets would mean depth entropy can pick good models without labels. A correlation near zero would falsify the claim. A further test is whether selecting the run with the lowest depth entropy recovers most of the gap between the median run and the oracle best run.

Real-image benchmarks. We use pretrained generators for FFHQ, AFHQv2, LSUN Cars, and SHHQ.

  • Edit purity is measured with off-the-shelf attribute classifiers that are used only for evaluation. For each edit we record the change in the target attribute and the summed change in all non-target attributes.
  • PIPL measures how straight and isolated each latent path is.
  • FID on edited images measures realism.
  • A matched user study measures meaningfulness.

Decisive results for the first claim. On SHHQ and LSUN Cars, where no published manual layer split exists, joint learning should beat both post-hoc masking and per-block SeFa on purity at equal PIPL. The learned bands on FFHQ should also roughly recover GANSpace's hand-chosen ranges without being told them.