Introduction
Modern generative models for image synthesis frequently rely on vector quantization to map continuous image data into a discrete sequence of tokens. While this approach simplifies the application of transformer architectures—originally designed for discrete language processing—it introduces significant information loss. In the context of semantic segmentation, where precise spatial and semantic boundaries are critical, this "quantization bottleneck" results in a noticeable degradation of detail.
The CAM-Seg (Continuous Autoregressive Model for Segmentation) framework addresses this limitation by transitioning from a discrete vocabulary to a continuous-valued embedding space. Preliminary analysis reveals that autoencoders utilizing quantized embeddings are approximately 8% less accurate in reconstructing segmentation masks compared to continuous-valued alternatives.
Methodology
CAM-Seg integrates a diffusion-guided transformer with a continuous Variational Autoencoder (VAE) to preserve fine-grained details. The architecture is composed of three primary modules:
- Continuous KL-VAE: Unlike VQ-VAEs that use a discrete codebook, CAM-Seg employs a KL-regularized VAE. This allows the model to map images and masks into a continuous latent space, maintaining high fidelity for thin or small objects such as poles and tires.
- Diffusion-Guided Transformer: To process continuous values, the framework utilizes a transformer that predicts a conditioning variable ($z$) rather than a discrete token. The transformer employs bidirectional attention, similar to Masked Autoencoders (MAE), to capture global context and long-range dependencies.
- Diffusion Bridge: A diffusion model serves as the bridge between the transformer's output and the continuous semantic embeddings. During training, it learns to map the conditioning variable $z$ to the target embedding. At inference, it denoises a random starting point into a clean semantic embedding guided by the transformer's context.
Experimental Results
The model was rigorously evaluated on urban datasets including Cityscapes, SemanticKITTI, ACDC, and CADEdgeTune.
Reconstruction and Robustness
The continuous KL-VAE achieved a reconstruction F1 score of 94.48%, a substantial improvement over the 86.32% achieved by the quantized VQ-VAE. Furthermore, the model demonstrated exceptional resilience to noise. It maintained approximately 95% of its baseline performance under Gaussian noise and remained robust against 50% "salt and pepper" noise, likely due to the inherent denoising nature of the diffusion component.
Zero-Shot Generalization
In zero-shot scenarios—testing on datasets or conditions not seen during training—CAM-Seg showed remarkable flexibility. On the ACDC dataset, which features adverse weather, the model achieved an Average Precision (AP) of 51.99 in fog. While performance was lower in heavy snow (40.69) and rain (43.50), the model still outperformed traditional quantized benchmarks by avoiding "blocky" artifacts and maintaining smoother semantic boundaries.
Conclusion
CAM-Seg demonstrates that maintaining continuity in the latent space is vital for high-fidelity semantic image generation. By leveraging diffusion as a regression and sampling mechanism, the model successfully adapts transformer architectures to continuous domains. Future work will focus on scaling the model to larger datasets and extending its capabilities to multi-task learning, including depth estimation and object detection.
Code Availability: https://github.com/mahmed10/CAMSS.git
Summary updated