GramStatTexNet: Efficient, Interpretable, and Neuro-Inspired Texture Analysis-by-Synthesis

Vasha DuTell, Zuyu Yun, Anne Harrington, Michael Y-s Fang, Mark Hamilton, Christian Kövesdi, Edward H. Adelson, William T. Freeman, Ruth RosenholtzarXiv 2026 · arXiv

TL;DR

GramStatTexNet (GSTN) is a hybrid texture synthesis framework that combines the interpretability of neuroscience-inspired Gabor filters with the power of Gramian-based correlation matching. It achieves visual quality comparable to deep learning models like Gatys et al. while using 30x fewer statistics, providing a transparent and biologically plausible window into how we perceive visual patterns.

What Problem Does This Paper Address?

Texture synthesis has long been split into two camps. On one side, classical neuroscience models (like the Portilla & Simoncelli model) use hand-curated statistics from multi-scale filter pyramids. These are interpretable and compact but often struggle to capture the high-fidelity realism of complex natural textures.

On the other side, deep learning approaches (like Gatys et al.) use the Gramian of VGG-Net features to achieve stunning realism. However, these models are "black boxes"—they are over-parameterized, computationally expensive, and their internal representations have no direct link to biological vision.

The authors of GramStatTexNet ask: Can we build a model that is as realistic as deep networks but as interpretable and efficient as classical neuroscience models?

Methods

The authors introduce GramStatTexNet (GSTN), an analysis-by-synthesis framework.

  1. The Filter Bank: Instead of learned convolutional layers, GSTN uses a fixed, multi-scale Gabor filter bank (a "pyramid") that mimics the receptive fields of neurons in the primary visual cortex (V1).
  2. The Statistics: It calculates the Gram matrix—the pairwise correlations between all filter responses across scales, orientations, and colors. This captures the "essence" of a texture without needing hand-picked statistical formulas.
  3. Synthesis: To create a new texture, the model starts with white noise and uses gradient descent to adjust pixel values until the noise image's Gramian matches the target texture's Gramian.
  4. Feature Importance Module: To understand which correlations actually matter, the authors use contrastive learning. They train a simple linear layer to compress the 28,929 statistics into a 100-dimensional vector that can distinguish between different texture categories.
Figure 1: Left: GramStatTexNet, our analysis-by-synthesis texture model, decomposes images into stacks (pyramids) of varying scales, orientations, and color channels, using a multi-scale pyramid filter bank. Pair-wise correlations between all pyramid images are represented in the Gram matrix; the upper triangle of this Gram matrix represents the full statistics set. Synthesis proceeds by calculating loss on statistics between a target and noise seed image, backpropagating this loss through the network to adjust pixel values until a synthesized image is produced. Right: To learn a reduced statistics set, a single fully connected layer compresses the Gram upper triangle to a reduced representation. These weights are trained using a contrastive loss, self-supervised by the objective of crops from the same texture image lying nearby in the reduced feature space.

Datasets

The model was evaluated using:

  • Describable Texture Dataset (DTD): A standard benchmark for static texture analysis.
  • DTD-141: A specific subset of 141 images used for rigorous quantitative comparison.
  • DynTex: A database of dynamic textures (videos) used to test the model's spatiotemporal extensions.

Key Insights and Ideas

  • The "Family" Taxonomy: Because the filters are known, every statistic in the Gramian has a "name" (e.g., "Cross-orientation" or "Cross-scale"). This allows researchers to perform ablation studies—dropping specific families of statistics to see exactly how they contribute to visual fidelity.
  • Efficiency via Contrastive Learning: The authors discovered that they could reduce the representation size by ~83% (down to 5,000 stats) and still achieve acceptable synthesis.
  • Discriminative vs. Generative: Interestingly, the statistics most important for identifying a texture (discriminative) aren't always the best for reconstructing it (generative). Randomly sampled subsets of statistics often outperformed the "top-ranked" ones for synthesis quality.
  • Unified Framework: The model isn't just for static images. By using 3D Gabor filters, it extends naturally to video (dynamic textures), and by using spatial pooling, it can simulate human peripheral vision (foveated synthesis).
Figure 4: Family-level texture fingerprints across DTD. Top: Hierarchical clustering on per-texture category mean signatures recovers a smoothness vs structure ordering without supervision; one representative exemplar per category at each leaf. Top heatmap: per-category mean of within-family z-scored stat values. Bottom heatmap: per-category within-family std of z-scored stats (centered; orange = above-average within-family heterogeneity, purple = below).

Key Results

  • Quality Parity: GSTN achieves synthesis quality competitive with deep-feature methods like Gatys et al., but with a 30x smaller representation (29k vs 852k statistics).
  • Pareto Frontier: In terms of the trade-off between representation size and perceptual quality, GSTN sits on the Pareto frontier, outperforming the classical Portilla & Simoncelli (P&S) model across almost all metrics.
  • Metric Performance: GSTN performed better than deep models on classical structural metrics (like SSIM and FSIM), while deep models still lead on metrics that use VGG features (like LPIPS), which is expected given their shared architecture.
Figure 3: Pareto comparison: representation size versus synthesis quality across five perceptual metrics on 141 DTD validation textures. GSTN-zpad sits on the Pareto frontier between P&S [43] and the order-of-magnitude-larger feature-space methods including Ustyuzhaninov [56] and Gatys [21], achieving competitive quality with 30× fewer statistics than Gatys on four of five metrics. On LPIPS, Gatys is superior, consistent with the metric evaluating similarity in the same VGG features that Gatys’ synthesis loss optimizes. Depletion variants (top-|w|, random, bottom-|w|) extend the GSTN frontier toward smaller representation sizes. µNCA not included due to compute limitations. See Appendix Figures 10 and 11. Error bars: 95% bootstrap confidence intervals on the mean over 141 textures, 1,000 resamples.

Limitations and Future Work

  • Fidelity Gap: While competitive, GSTN syntheses are still visibly lower-fidelity than state-of-the-art diffusion models.
  • Human Validation: The current study relies on algorithmic metrics; the authors note that future work should include psychophysical testing with human observers to see if these "metamers" truly fool the human eye.
  • Computational Tractability: The contrastive compression technique is currently only tractable for the spatial model; scaling this to the much larger spatiotemporal (video) statistics set remains a challenge.
  • Integration with Generative AI: The authors suggest that GSTN statistics could eventually be used as guidance for diffusion-based generative models.

Summary updated

Abstract

Deep networks achieve high realism in texture synthesis but often rely on opaque, over-parameterized representations that have limited direct connection to biological vision. Neuroscience-informed pyramid models are interpretable, compact, and biologically inspired, but rely on small, hand-curated statistics sets that are difficult to generalize or extend. We introduce GramStatTexNet, a hybrid analysis-by-synthesis framework that combines the multi-scale Gabor filter structure of classical models with the flexibility of Gramian-based correlations, organized into structured filter families. This structured representation enables analyses that are challenging for deep-feature methods: each statistic carries a named filter-family identity, allowing us to decompose synthesis fidelity into per-family contributions and to learn an interpretable, low-dimensional embedding that preserves perceptual texture content. We compare our framework to classical and deep texture-synthesis baselines across diverse texture categories, and on multiple perceptual metrics, highlighting the compactness and interpretability advantages of this hybrid representation, alongside competitive synthesis quality. This same framework extends naturally to peripheral textures via spatial pooling, and to dynamic textures using a spatiotemporal filter bank. GramStatTexNet provides a unified, interpretable framework for analyzing and modeling visual information with natural extensions across space, time, and eccentricity. Project page: https://vdutell.github.io/gstn
BibTeX
@misc{dutell2026gramstattexnet,
  author = {Vasha DuTell and Zuyu Yun and Anne Harrington and Michael Y-S Fang and Mark Hamilton and Christian Kövesdi and Edward H. Adelson and William T. Freeman and Ruth Rosenholtz},
  title = {{GramStatTexNet: Efficient, Interpretable, and Neuro-Inspired Texture Analysis-by-Synthesis}},
  year = {2026},
  eprint = {2611.23755},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.23755},
}
Figure