Federated Dataset Simulation: Inducing Label-Free Heterogeneity Across Tasks

Vasilis Siomos, Lam Ngo, Jonathan Passerat-Palmbach, Giacomo TarroniarXiv 2026 · arXiv

TL;DR

Researchers have introduced Federated Dataset Simulation (FDS), a framework that moves beyond simple label-based data partitioning to create more realistic, "label-free" heterogeneity for Federated Learning (FL). By using foundation model embeddings and vision-language models, their open-source tool fds induces complex feature-space diversity that challenges FL algorithms more effectively than traditional methods—even for tasks like image segmentation and object detection where labels aren't easily partitioned.

What Problem Does This Paper Address?

Federated Learning (FL) is built on the reality that data is "non-IID" (not identically and independently distributed) across different users. However, most researchers evaluate their FL algorithms by taking a centralized dataset (like CIFAR) and artificially splitting it using a method called Dirichlet label skew.

The authors argue this is a poor proxy for the real world for two reasons:

  1. It’s one-dimensional: It only accounts for which classes a client sees, ignoring how those classes look (e.g., one hospital might have high-res X-rays while another has low-res ones).
  2. It requires labels: This makes it impossible to simulate federated versions of "dense" tasks like semantic segmentation or object detection, where a single image contains multiple objects and labels.

As a result, FL benchmarks are often too easy or don't generalize to the complex, multi-dimensional heterogeneity found in actual production deployments.

Methods

The authors formalize the problem as Federated Dataset Simulation (FDS) and decompose it into two parts: a representation function ($\Phi$) that maps data into a latent space, and an assignment mechanism ($\pi$) that allocates that data to clients.

They propose two novel, label-free instantiations of this framework:

  • Geometric Partitioning: This method uses a self-supervised foundation model (like DINOv2) to create embeddings of the images. It then uses k-means clustering to group visually similar images into "clients." This induces "covariate shift"—clients end up specializing in specific visual domains or scene compositions without ever looking at a label.
  • Semantic Partitioning: This pipeline uses a Vision-Language Model (VLM) and an LLM to "tag" images with human-interpretable criteria (e.g., "lighting," "cuisine type," or "location"). A product-of-Dirichlets mechanism then assigns images to clients based on these semantic traits. This allows researchers to say, "Client A mostly sees outdoor scenes at night," providing a controllable and explainable way to induce heterogeneity.

Datasets

The study evaluates these methods across four diverse vision datasets:

  • Classification: Tiny-ImageNet and Food101.
  • Semantic Segmentation: ADE20K.
  • Object Detection: PASCAL VOC.

They also compare their simulated partitions against naturally federated datasets like Fed-ISIC2019 (skin lesions), FEMNIST (handwriting), and Office-Caltech10 to ensure their simulations match the "intensity" of real-world data splits.

Key Insights and Ideas

  • Labels aren't everything: Surprisingly, Geometric Partitioning (which uses no labels) often induces higher label divergence than methods specifically designed to skew labels. By grouping images that look alike, you naturally group images that share the same category.
  • The "Aha" on Optimization: The authors show that feature-space heterogeneity is a better predictor of FL difficulty than label-space heterogeneity. A model might struggle more with a "Geometric" split because it forces the model to learn from highly biased visual features, making global convergence much harder.
  • Dense Tasks are no longer "second-class": By providing a principled way to partition segmentation and detection datasets, the authors open the door for FL research in more complex, high-stakes domains like medical imaging and autonomous driving.
  • Task Relevance Matters: Not all heterogeneity is equal. Splitting data by "lighting" might not hurt a model's performance much, but splitting by "main object type" (high mutual information with the task) creates a massive optimization challenge.

Key Results

  • Higher Stress Test: In almost all configurations, Geometric partitioning resulted in the lowest global validation performance. For example, on Tiny-ImageNet with 10 clients, Geometric partitioning dropped FedAvg accuracy by ~9.8% compared to a random split, significantly more than the ~3.1% drop caused by standard label-skew.
  • Ranking Flips: The authors found that the "best" FL aggregation algorithm can change depending on the partitioning method used. This suggests that many current FL papers might be over-optimizing for the specific (and limited) challenge of label skew.
  • Realism: Geometric partitioning was shown to match or exceed the feature-space heterogeneity found in "natural" federated datasets like FEMNIST and Office-Caltech10.

Limitations and Future Work

The authors identify several areas for growth:

  • Modality: The current work is focused on computer vision. Extending FDS to text, tabular, or multi-modal data is a clear next step.
  • Acquisition Factors: While the methods capture "content" and "scene" well, they don't explicitly model "acquisition factors" like specific camera sensor noise or compression artifacts, which are common in real-world FL.
  • VLM Reliability: The Semantic Partitioning pipeline is only as good as the underlying VLM. The authors noted that current models still struggle with "enumeration" (counting objects), which limits the types of semantic criteria that can be used.

The researchers have released their work as a modular Python package, fds, to encourage the community to adopt more rigorous and diverse evaluation standards.

Summary updated

Abstract

Federated Learning enables collaborative model training across decentralized clients without sharing raw data. While the field has seen rapid growth, the datasets underpinning FL evaluation have not kept pace: real federated datasets remain scarce, and most studies rely on small-scale centralized classification datasets partitioned via Dirichlet label skew, a one-dimensional proxy for the multi-dimensional heterogeneity present in real federations. Moreover, label-based partitioning is by construction inapplicable to non-classification tasks such as segmentation and detection, for which no principled simulation method exists. We cast the conversion of centralized datasets into federated benchmarks as the task of Federated Dataset Simulation (FDS), and decompose it into a representation function $Φ$ and an assignment mechanism $π$. We provide two label-free instantiations -- geometric partitioning via pretrained model embeddings and semantic partitioning via vision-language model-derived criteria -- applicable to classification and dense prediction tasks alike. We evaluate across four datasets spanning classification, segmentation, and detection, and show that both methods induce heterogeneity that matches or exceeds that of label-skew partitioning, with measurable impact on downstream FL performance. We release fds, a modular open-source package for end-to-end federated dataset simulation, heterogeneity measurement, and FL training, at https://github.com/siomvas/fds.
BibTeX
@misc{siomos2026federated,
  author = {Vasilis Siomos and Lam Ngo and Jonathan Passerat-Palmbach and Giacomo Tarroni},
  title = {{Federated Dataset Simulation: Inducing Label-Free Heterogeneity Across Tasks}},
  year = {2026},
  eprint = {2611.19975},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.19975},
}
Figure