CAM-Seg: A Continuous-valued Embedding Approach for Semantic Image Generation

Masud Ahmed, Zahid Hasan, Syed Arefinul Haque, Abu Zaher Md Faridee, Sanjay Purushotham, Suya You, Nirmalya RoyarXiv 2025 · DOI · arXiv

Introduction

Modern generative models for image synthesis frequently rely on vector quantization to map continuous image data into a discrete sequence of tokens. While this approach simplifies the application of transformer architectures—originally designed for discrete language processing—it introduces significant information loss. In the context of semantic segmentation, where precise spatial and semantic boundaries are critical, this "quantization bottleneck" results in a noticeable degradation of detail.

The CAM-Seg (Continuous Autoregressive Model for Segmentation) framework addresses this limitation by transitioning from a discrete vocabulary to a continuous-valued embedding space. Preliminary analysis reveals that autoencoders utilizing quantized embeddings are approximately 8% less accurate in reconstructing segmentation masks compared to continuous-valued alternatives.

Comparison of Quantized vs. Continuous Embeddings and Ground Truth

Methodology

CAM-Seg integrates a diffusion-guided transformer with a continuous Variational Autoencoder (VAE) to preserve fine-grained details. The architecture is composed of three primary modules:

  1. Continuous KL-VAE: Unlike VQ-VAEs that use a discrete codebook, CAM-Seg employs a KL-regularized VAE. This allows the model to map images and masks into a continuous latent space, maintaining high fidelity for thin or small objects such as poles and tires.
  2. Diffusion-Guided Transformer: To process continuous values, the framework utilizes a transformer that predicts a conditioning variable ($z$) rather than a discrete token. The transformer employs bidirectional attention, similar to Masked Autoencoders (MAE), to capture global context and long-range dependencies.
  3. Diffusion Bridge: A diffusion model serves as the bridge between the transformer's output and the continuous semantic embeddings. During training, it learns to map the conditioning variable $z$ to the target embedding. At inference, it denoises a random starting point into a clean semantic embedding guided by the transformer's context.
The CAM-Seg training and inference pipelines

Experimental Results

The model was rigorously evaluated on urban datasets including Cityscapes, SemanticKITTI, ACDC, and CADEdgeTune.

Reconstruction and Robustness

The continuous KL-VAE achieved a reconstruction F1 score of 94.48%, a substantial improvement over the 86.32% achieved by the quantized VQ-VAE. Furthermore, the model demonstrated exceptional resilience to noise. It maintained approximately 95% of its baseline performance under Gaussian noise and remained robust against 50% "salt and pepper" noise, likely due to the inherent denoising nature of the diffusion component.

Zero-Shot Generalization

In zero-shot scenarios—testing on datasets or conditions not seen during training—CAM-Seg showed remarkable flexibility. On the ACDC dataset, which features adverse weather, the model achieved an Average Precision (AP) of 51.99 in fog. While performance was lower in heavy snow (40.69) and rain (43.50), the model still outperformed traditional quantized benchmarks by avoiding "blocky" artifacts and maintaining smoother semantic boundaries.

Conclusion

CAM-Seg demonstrates that maintaining continuity in the latent space is vital for high-fidelity semantic image generation. By leveraging diffusion as a regression and sampling mechanism, the model successfully adapts transformer architectures to continuous domains. Future work will focus on scaling the model to larger datasets and extending its capabilities to multi-task learning, including depth estimation and object detection.

Code Availability: https://github.com/mahmed10/CAMSS.git

Summary updated

Abstract

Traditional transformer-based semantic segmentation relies on quantized embeddings. However, our analysis reveals that autoencoder accuracy on segmentation mask using quantized embeddings (e.g. VQ-VAE) is 8% lower than continuous-valued embeddings (e.g. KL-VAE). Motivated by this, we propose a continuous-valued embedding framework for semantic segmentation. By reformulating semantic mask generation as a continuous image-to-embedding diffusion process, our approach eliminates the need for discrete latent representations while preserving fine-grained spatial and semantic details. Our key contribution includes a diffusion-guided autoregressive transformer that learns a continuous semantic embedding space by modeling long-range dependencies in image features. Our framework contains a unified architecture combining a VAE encoder for continuous feature extraction, a diffusion-guided transformer for conditioned embedding generation, and a VAE decoder for semantic mask reconstruction. Our setting facilitates zero-shot domain adaptation capabilities enabled by the continuity of the embedding space. Experiments across diverse datasets (e.g., Cityscapes and domain-shifted variants) demonstrate state-of-the-art robustness to distribution shifts, including adverse weather (e.g., fog, snow) and viewpoint variations. Our model also exhibits strong noise resilience, achieving robust performance ($\approx$ 95% AP compared to baseline) under gaussian noise, moderate motion blur, and moderate brightness/contrast variations, while experiencing only a moderate impact ($\approx$ 90% AP compared to baseline) from 50% salt and pepper noise, saturation and hue shifts. Code available: https://github.com/mahmed10/CAMSS.git
BibTeX
@misc{ahmed2025cam,
  author = {Masud Ahmed and Zahid Hasan and Syed Arefinul Haque and Abu Zaher Md Faridee and Sanjay Purushotham and Suya You and Nirmalya Roy},
  title = {{CAM-Seg: A Continuous-valued Embedding Approach for Semantic Image Generation}},
  year = {2025},
  eprint = {2503.15617},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2503.15617},
}
Figure