DenseTwins: Dense Modality Decoupling for Multimodal Self-Supervision

Carlo Sgaravatti, Daniele Kopyshevskiy, Davide Secco, Giacomo BoracchiarXiv 2026 · arXiv

TL;DR

DenseTwins is a new self-supervised learning (SSL) framework that learns high-resolution, pixel-level representations from multimodal data (like RGB and LiDAR). By explicitly splitting its latent space into "shared" and "modality-specific" components, it captures both the common semantics and the unique details of each sensor, setting new performance benchmarks for dense tasks like semantic segmentation and object detection.

What Problem Does This Paper Address?

In fields like autonomous driving and robotics, we rely on multiple sensors—cameras, LiDAR, thermal imaging—to understand the world. While these modalities provide complementary information, training models to use them typically requires massive, expensive human-annotated datasets.

Self-supervised learning (SSL) offers a way to learn from unlabeled data, but existing multimodal SSL methods have two major drawbacks:

  1. Alignment Overkill: Many methods (like CLIP) try to force different modalities to look identical in the latent space. This ignores "modality-specific" cues, such as the unique texture in an RGB image or the precise geometry in a depth map.
  2. Global Bias: Most current frameworks learn a single global vector for an entire image. This is fine for classifying a whole scene but fails miserably for "dense" tasks like semantic segmentation or object detection, where spatial structure is everything.

DenseTwins bridges this gap by learning dense, spatially-aware representations that respect the unique contributions of each sensor.

Methods

DenseTwins builds on the principles of redundancy reduction (specifically the Barlow Twins objective) and modality decoupling.

The architecture consists of two separate encoders (one per modality) that feed into projection heads. The key innovation is the structure of the resulting embedding space:

  • Modality-Shared Subspace: Captures information common to both sensors (e.g., "there is a car here").
  • Modality-Specific Subspace: Preserves information unique to one sensor (e.g., the car's color in RGB or its 3D shape in LiDAR).
Figure 1. The structure of the dense embeddings of DenseTwins. Two separate encoders F1, F2 process the two input modalities X1, X2 separately, producing two feature maps F1, F2. Finally, two projection heads project F1, F2 into a common latent space Z1, Z2, each divided into modality-shared S1, S2 and modality-specific U1, U2 components.

To train this, the authors use a multi-part loss function:

  • Intra-modal Loss: Ensures consistency between different augmented views of the same modality.
  • Cross-modal Loss: Aligns the shared subspaces across modalities while explicitly decorrelating the modality-specific subspaces to prevent them from collapsing into the shared representation.
  • Dense Correspondence: Instead of assuming pixels align perfectly, the model uses a feature-similarity search to find semantically matching patches between modalities before applying the loss. This makes the system robust to sensor misalignment.

Datasets

The authors evaluated DenseTwins across an impressive array of six multimodal sensing pairs and seven datasets:

  • RGB–Depth: SUN-RGBD
  • RGB–Event: EventScape and DSEC
  • RGB–LiDAR: Semantic-KITTI
  • RGB–DEM (Digital Elevation Models): GeoNRW
  • SAR–Optical: BigEarthNet
  • RGB–Thermal: KAIST

Key Insights and Ideas

  • Decoupling is Essential: The paper demonstrates that forcing modalities to be perfectly aligned actually hurts performance. By allowing the model to "keep" modality-specific information in a dedicated subspace, the resulting features are much more expressive for downstream tasks.
  • Dense Matching > Positional Matching: By matching features based on similarity rather than fixed pixel coordinates, DenseTwins remains effective even when sensors aren't perfectly calibrated or aligned.
  • Redundancy Reduction for Separation: While redundancy reduction is usually used to make features similar, the authors show it can be flipped (via a decorrelation objective) to ensure modality-specific features stay distinct.
Figure 2. DenseTwins architecture. For each modality m ∈{1, 2}, two augmented views are processed by a modality-specific backbone Fm, followed by a projection head Pm that produces both global and dense embeddings. The embedding space is explicitly partitioned into a shared subspace (red) and a modality-specific subspace (blue). Training is based on dense correspondences between feature vectors within each modality and across modalities computed using cosine similarities. The shared components are aligned through redundancy reduction and invariance objectives, while modality-specific components are decorrelated across modalities to preserve complementary information. Intra-modal learning, instead, applies a redundancy-reduction objective, encouraging invariance between augmented views. The final objective combines intra-modal and cross-modal losses applied to both global and dense representations.

Key Results

DenseTwins consistently outperformed or matched state-of-the-art multimodal SSL baselines (like DeCUR and CLIP) and single-modal methods (like DINO and iBOT).

  • Semantic Segmentation: On the SUN-RGBD benchmark, DenseTwins achieved 38.2 mIoU with only 10% labels, significantly outperforming the baseline "None" (35.7) and other SSL methods.
  • Object Detection: In autonomous driving scenarios (KAIST and DSEC), DenseTwins achieved the best overall AP50 scores, proving that dense pretraining is superior to global pretraining for locating objects.
  • Frozen Probing: When the backbone is "frozen" (not fine-tuned), DenseTwins shows a massive lead. On SUN-RGBD, its FCN score was 40.6, compared to DeCUR's 35.8 and SimCLR's 30.4.

Limitations and Future Work

  • Manual Partitioning: Currently, the ratio of shared vs. specific dimensions ($K_s/K_u$) must be set manually. The authors found that the optimal ratio varies by modality (e.g., 75% shared for RGB-Depth vs. 62% for RGB-DEM). Automating this split is a suggested next step.
  • Computational Cost: The high-dimensional projection heads used during training increase training time by about 25% and peak memory by 26%. However, these heads are discarded after training, so there is no penalty during actual use (inference).
  • Spatial Modalities Only: The current formulation relies on spatial organization, meaning it cannot yet incorporate non-spatial modalities like text.

Summary updated

Abstract

Multimodal perception is central to many computer vision applications, including robotics and autonomous driving, where different sensing modalities provide complementary cues. Self-supervised learning offers a scalable solution for exploiting large amounts of unlabeled multimodal data. However, existing methods either neglect modality-specific information by enforcing feature-space alignment between modalities or learn only global representations without preserving spatial structure. We propose DenseTwins, a multimodal self-supervised framework that learns dense representations by explicitly decoupling the dense feature space into shared and modality-specific components. The proposed approach combines dense cross-modal correspondences with redundancy-reduction objectives, enabling the model to capture both shared semantics and modality-specific cues within a structured dense embedding space. DenseTwins jointly optimizes an intra-modal objective across augmented views and a cross-modal objective that aligns shared features while decorrelating modality-specific information. We evaluate DenseTwins across multiple multimodal benchmarks and sensing setups, achieving state-of-the-art or competitive performance across multimodal self-supervised baselines. Code available at: https://github.com/CarloSgaravatti/DenseTwins.
BibTeX
@misc{sgaravatti2026densetwins,
  author = {Carlo Sgaravatti and Daniele Kopyshevskiy and Davide Secco and Giacomo Boracchi},
  title = {{DenseTwins: Dense Modality Decoupling for Multimodal Self-Supervision}},
  year = {2026},
  eprint = {2611.04710},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.04710},
}
Figure