Camera networks increasingly mix ordinary color cameras with thermal or near-infrared cameras that keep working at night. Matching the same person across the two spectra is called visible–infrared person re-identification. Every strong method for it assumes one training machine can see color and infrared images of the same people side by side. Privacy rules push the other way. Sites that record people usually cannot pool footage, and in practice a given site often runs only one kind of camera. Under federated learning, where each site trains locally and a server averages the model weights, the cross-spectral problem changes character. The gap between color and infrared features still forms, but no single participant can see it, so no participant can compute the losses that normally close it.
We propose to measure that gap where it can be measured, at the server. Only per-modality summary statistics of the learned features are sent there, never images or identities. From them the server builds a closed-form "feature translator" that maps color-camera embeddings into the infrared embedding distribution and back. Each site then trains its encoder so that identity survives this translation, and so that its modality's distance from the other one moves toward a calibrated, non-zero target rather than toward zero. The two-sentence version is this. Cross-modality translation moves out of pixel space, which a single-modality site cannot learn, and into a server-estimated map between feature distributions. That map lets modality-pure sites train against a spectrum they have never observed.
Core Claim
In a federated setting where every client holds images from only one spectrum and identity labels are local, the standard cross-modal losses cannot be computed. These are angular triplets (Bi-directional Exponential Angular Triplet Loss for RGB-Infrared Person Re-Identification), progressive grayscale-to-infrared stages (Learning Progressive Modality-Shared Transformers for Effective Visible-Infrared Person Re-identification), and generated paired images (Cross-Modality Paired-Images Generation for RGB-Infrared Person Re-Identification). We claim that a server-side translator estimated from aggregated first- and second-order feature statistics carries enough cross-modal signal to recover a substantial part of the centralized accuracy. Combined with a loss that asks identity to survive translation, it should beat both naive federated averaging and cheap pixel-level fixes. We further claim that driving the measured gap to zero is worse than stopping at a calibrated non-zero gap. That prediction comes from Mind the Gap, which showed the gap is a geometric byproduct of separate encoder pathways and contrastive training, and that tuning rather than eliminating it improved zero-shot accuracy and fairness in CLIP.
Why This Gap Is Real
The file contains federated person re-identification only for color imagery. Optimizing Performance of Federated Person Re-identification (FedReID) shares a backbone, keeps identity heads local, and clusters and weights clients. It shows naive averaging loses 1.3% Rank-1 on MSMT17 relative to isolated training, while its full method gains 4.1%. It never mixes spectra. That paper motivates the proposal and supplies our aggregation skeleton.
The federated translation papers constrain the design. FedMed-GAN and FedMed-ATL train CycleGAN-style translators by averaging generators across hospitals. Each hospital, however, holds both MRI contrasts, unpaired. Efficient Federated Domain Translation reports that federated GANs fail when clients cannot inspect source and target domains together, and falls back to server-side one-dimensional barycenters. Our modality-pure setting is the harsh case Zhou et al. describe. We take their lesson, that the server should hold distributional summaries, and apply it to feature space instead of pixels.
The closest federated multimodal work is FDARN. It splits features into globally shared modality-agnostic parts and locally kept modality-specific parts, with an angular modality discriminator. It does not translate, does not target retrieval across spectra, and treats the gap as something to remove. FedMAC imputes missing ECG channels in latent space, but within clients that hold both. FedHealth matches second-order feature statistics (CORAL) between source and target within one sensor type. Our translator is mathematically a relative of CORAL. The differences are that it is estimated across clients, applied across spectra, and used inside a training loss rather than only as an alignment penalty. The ablations isolate that difference.
Research Program
Each client $k$ holds images of one modality $m_k \in \{V, I\}$ (visible or infrared) and locally defined identity labels. Following FedPav, it shares an encoder $f$ and keeps a private identity classifier $h_k$. Each round, a client uploads its encoder update together with the mean and covariance of its L2-normalized embeddings. These are summed across clients of the same modality under secure aggregation, with optional Gaussian noise for differential privacy. The server combines them into $\mu_V, \Sigma_V, \mu_I, \Sigma_I$ and builds the translator
$$T_{V\to I}(z) = \Sigma_I^{1/2}\,\Sigma_V^{-1/2}\,(z-\mu_V) + \mu_I ,$$
with $T_{I\to V}$ defined symmetrically. Here $z$ is an embedding, and a low-rank-plus-diagonal covariance keeps communication small. The server broadcasts the translator and the other modality's statistics with the next model.
A visible-camera client with embedding $z=f(x)$ and label $y$ then minimizes
$$\mathcal{L}_k = \ell_{\text{id}}(h_k(z),y) + \alpha\,\ell_{\text{id}}\big(h_k(T_{V\to I}(z)),y\big) + \beta\,\big(\lVert \bar z_k - \mu_I\rVert - g^\star\big)^2 .$$
The first term is ordinary identity loss. The second, the translation-consistency term, requires the identity to remain recognizable after the embedding is moved into the infrared distribution. This pushes identity information into directions the modality shift leaves intact, a feature-space analogue of the geometric consistency Arar et al. enforce so that translation cannot "absorb" what should be preserved. The third term pulls the client's mean embedding $\bar z_k$ toward a target distance $g^\star$ from the infrared centroid, instead of toward zero.
$g^\star$ is chosen from a small grid on a held-out validation client, and we report the full accuracy-versus-gap curve rather than one tuned point. We also track the measured gap, the distance between modality centroids of normalized embeddings as defined in Mind the Gap, over training rounds. This tells us whether the translator actually shrinks the gap and where accuracy peaks.
Three analyses accompany the method. The first is a leakage study: we attempt to reconstruct images or identities from the uploaded statistics, compared with raw gradient uploads. The second measures communication overhead per round. The third tests whether the gap persists even with a single shared encoder that uses no modality-specific layers, which Mind the Gap's cone-effect argument predicts, weakly, from separate pathways alone.
Baselines
The naive reference point is FedAvg over a single shared encoder and classifier, with no modality handling. It shows how badly modality-pure partitioning hurts. It will be run with the same backbone, number of rounds, local epochs, and hyperparameter search budget as every other federated method. Every federated method gets an equal grid over learning rate and local epochs, and all use the same client splits and seeds.
The strongest existing federated method is FedPav with client clustering and cosine-distance weighting from FedReID. We port it unchanged to the cross-spectral splits. We also run a strengthened version that adds the Deep Zero-Padding modality indicator from RGB-Infrared Cross-Modality Person Re-identification and the expAT angular triplet loss computed within each client. Beating it means higher cross-modal Rank-1 and mean average precision (mAP) on both benchmarks across three seeds, by more than the seed spread. No reported result was found on Lacuna for any federated visible–infrared re-identification method, so we establish these numbers ourselves.
The centralized reference ceiling is PMT, which reports 67.53% Rank-1 and 64.98% mAP on SYSU-MM01 all-search and 84.83% Rank-1 on RegDB visible-to-thermal (PMT). The older centralized expAT reports 38.57% Rank-1 and 38.61% mAP on SYSU-MM01. We will retrain centralized versions on our backbone so the ceiling is measured under matched compute, and report the fraction of the centralized-minus-FedAvg gap each federated method recovers.
The ablations isolate the key ingredient. Mean shift only drops the covariance terms. Alignment only replaces the consistency term with a CORAL-style penalty that pulls modality statistics together. Zero-gap target sets $g^\star=0$. No consistency term keeps only the gap penalty.
Why not just…?
Why not just convert the color images to grayscale and run FedAvg? Grayscale is a free, local pixel translation available to every visible-light client, and PMT showed grayscale as a stepping stone alone gave more than 5% Rank-1 centrally. This is a real threat. We run it, with and without zero-padding indicators and local modality-specific normalization. If it closes most of the federated gap, our contribution shrinks to a diagnosis. It does nothing for infrared clients, though, and it removes color only, not the reflected-versus-emitted intensity differences. We therefore expect a residual gap, measurable in the centroid distance, that feature-space translation targets. We will also run our method on top of grayscale inputs, so that the two effects are separated rather than pitted against each other.
Why not just train a federated CycleGAN to produce pseudo-infrared images, as FedMed-GAN does? In our setting no client holds both modalities, so no client can form the cycle, and a visible-only client cannot train an infrared discriminator. Efficient Federated Domain Translation reports that federated GANs fail in exactly this configuration. FedMed-GAN's success relied on clients holding both contrasts. We will still implement a split version: generator averaged, discriminators local at infrared clients, generator updates relayed through the server. That settles the tension empirically, reporting both re-identification accuracy and the cost of shipping generators.
Why not just align second-order statistics, as FedHealth does with CORAL? This is the alignment-only ablation. CORAL-style alignment drives the modalities to overlap, and Mind the Gap predicts overlap is not the optimum. It also constrains only the statistics, not whether identity information survives the shift. If alignment-only matches the full method, the consistency term and calibrated gap are unnecessary, and we will report that plainly.
Why not just keep modality-specific normalization layers local and average the rest? Local normalization absorbs first- and second-order shifts per client, and expAT's CSBN shows normalization choices matter here. It is included in the strengthened baseline. It cannot tell a visible-light client anything about infrared statistics, however, which the broadcast translator does. The comparison with and without local normalization, under our loss, shows whether the two are redundant.
Proposed Experiments
On SYSU-MM01, cameras are natural clients: four visible-light cameras and two near-infrared cameras become six modality-pure clients with camera-local labels. We evaluate the standard all-search and indoor-search protocols on held-out identities. On RegDB, whose images are paired visible and thermal, we assign visible images to visible clients and thermal images to thermal clients. We vary the identity overlap between the two sides from zero to full, since overlap governs how much the modality statistics describe the same people. Metrics are Rank-1 and mAP in both query directions, the measured gap per round, uplink bytes per round, and accuracy under differential-privacy noise on the uploaded statistics.
The decisive checks are three. First, the full method must beat strengthened FedPav and the grayscale baseline on both datasets across seeds. Second, the accuracy-versus-$g^\star$ curve must peak at a non-zero gap, or the Mind the Gap prediction fails in this setting. Third, removing the covariance or consistency terms must cost accuracy, or the gain came from something simpler. If the method helps only at high identity overlap on RegDB, that would show the translator is exploiting shared people rather than shared spectra, and we would state that limit directly.
References
- Bi-Directional Exponential Angular Triplet Loss for RGB-Infrared Person Re-Identification
- Learning Progressive Modality-Shared Transformers for Effective Visible-Infrared Person Re-identification
- Cross-Modality Paired-Images Generation for RGB-Infrared Person Re-Identification
- Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learning
- Optimizing Performance of Federated Person Re-identification: Benchmarking and Analysis
- FedMed-GAN: Federated domain translation on unsupervised cross-modality brain image synthesis
- FedMed-ATL: Misaligned Unpaired Cross-Modality Neuroimage Synthesis via Affine Transform Loss
- Efficient Federated Domain Translation
- Cross-Modal Federated Human Activity Recognition via Modality-Agnostic and Modality-Specific Representation Learning
- FedMAC: Tackling Partial-Modality Missing in Federated Learning with Cross-Modal Aggregation and Contrastive Regularization
- FedHealth: A Federated Transfer Learning Framework for Wearable Healthcare
- Unsupervised Multi-Modal Image Registration via Geometry Preserving Image-to-Image Translation
- RGB-Infrared Cross-Modality Person Re-identification