Bg-Mix: On Using Image Backgrounds for Data Augmentation in Electron Microscopy Images

Ankur Srivastava, Joana Sarah Grah, Katharina Ladewig, Nils Körber

TL;DR

Bg-Mix is a specialized data augmentation technique for electron microscopy (EM) that creates new training samples by mixing foreground virus particles from one image with backgrounds extracted from another. By leveraging the complex, "content-rich" backgrounds inherent in EM data, it improves object detection performance by up to 3.3 mAP points, helping deep learning models overcome the challenge of small, sparsely labeled biomedical datasets.

What Problem Does This Paper Address?

In virology, Electron Microscopy (EM) is the gold standard for visualizing pathogens, but analyzing these images is a labor-intensive task requiring significant expert knowledge. While deep learning could automate this process, researchers face a major hurdle: data scarcity.

Biomedical imaging datasets are expensive to acquire and time-consuming to annotate. A typical EM dataset might only contain a few hundred images. Furthermore, EM images have "content-rich" backgrounds filled with cell debris and other materials that can easily be confused with viruses. Standard data augmentation techniques often rely on massive external datasets (like COCO) which don't match the specific visual characteristics of microscopy. The authors set out to find a way to maximize the utility of the limited data already available by better utilizing these complex backgrounds.

Methods

The authors propose Bg-Mix, a method that decomposes training images into their constituent parts—foreground objects (viruses) and backgrounds—and then rearranges them to form novel training samples.

Figure 1: Bg-Mix. The background of a random training image (a) and the foreground objects of another random training image (b) are extracted and blended to form the background-augmented image (c).

The process involves three main steps, for which the authors tested several variations:

  1. Background Extraction: How do we get an "object-free" background?
  2. Method A: Selecting the largest empty square in an image and resizing it.
  3. Method B: Using segmentation masks to "zero out" (black out) the pixels where viruses were.
  4. Method C: Biharmonic Inpainting, which fills the holes left by removed viruses with smooth interpolations of the surrounding texture.
  5. Object Blending: How do we paste the viruses onto the new background?
  6. Hard edges: Pasting them exactly as they are.
  7. Blurred edges: Using a small Gaussian kernel to soften the boundaries.
  8. Object Placement: Where do the viruses go?
  9. The authors tested six strategies, ranging from keeping objects in their original coordinates to randomly placing objects from multiple different images or classes onto a single background.

Datasets

The study evaluated Bg-Mix on two publicly available negative-staining EM datasets:

  • EMViR: 1,084 Scanning-Transmission EM (STEM) images featuring four classes of respiratory viruses (Adeno, Corona, Orthomyxo, and Paramyxo).
  • VISN: 622 Transmission EM (TEM) images featuring six virus classes (including SARS-CoV-2 and Herpes simplex).

Both datasets are characterized by having only one virus class per image, which allowed the researchers to experiment with "class-matched" backgrounds.

Key Insights and Ideas

  • Backgrounds are Signal, Not Just Noise: In EM, the background isn't just empty space; it contains debris and cellular structures that the model must learn to distinguish from viruses. By recombining real backgrounds with different foregrounds, Bg-Mix forces the model to become more robust to these distractors.
  • Inpainting is Superior: The ablation study revealed that biharmonic inpainting (Method C) generally performed best. Unlike simple zeroing, inpainting maintains the textural continuity of the background, creating more realistic synthetic images.
  • Simplicity Wins in Placement: Interestingly, the best-performing configuration (Run 5 in the ablation study) involved pasting objects at their original positions onto an inpainted background from a different image. This suggests that the spatial distribution of particles in the original samples carries important domain-specific information that random placement might disrupt.
  • Domain-Specific beats General-Purpose: Bg-Mix consistently outperformed or matched the popular "Copy-Paste" augmentation method, which often uses backgrounds from natural image datasets that look nothing like an electron microscope's output.

Key Results

The researchers tested Bg-Mix using two popular architectures: Mask R-CNN and RTMDet-Ins.

Table 1: Mask R-CNN and RTMDet-Ins models with Copy-Paste and Bg-Mix on EMViR and VISN datasets. The brackets show the change from the first row of the same block. p is the probability with which Copy-Paste or Bg-Mix was applied to a training image (p=0.5 for EMViR; p=0.01 for VISN). Best results are shown in bold.
  • Significant mAP Gains: Bg-Mix provided an increase of +0.8 to +3.3 box mAP points over the baseline across both datasets.
  • EMViR Success: The largest gain (+3.3 mAP) was seen on the EMViR dataset using the RTMDet-Ins architecture.
  • Ablation Highlights: The detailed ablation study (Table 2) showed that while many combinations help, the "sweet spot" (Run 5) used inpainting, hard edges, and original placement to reach a peak of 58.3 mAP on EMViR.
Table 2: Ablation study of Bg-Mix on the EMViR dataset with Mask R-CNN: all combinations of background extraction, blending and object placement methods. All runs except baseline use p = 0.5. Best setup is shown in bold.

Limitations and Future Work

While effective, the study has a few limitations:

  • Domain Specificity: The current evaluation is limited to virus detection in negative-staining EM images. While the authors believe the findings are transferable to other "content-rich" background modalities (like other types of microscopy), this hasn't been tested yet.
  • Class Diversity: The datasets used only had one class per image. Future work could explore how Bg-Mix handles more complex scenes where multiple different types of objects naturally co-occur.
  • Saturated Baselines: On the VISN dataset, the baseline performance was already quite high (77.7 mAP), which led to more modest gains (+0.8 mAP) compared to the more challenging EMViR dataset.

Summary updated

Abstract

Electron microscopy (EM) serves as a cornerstone in virology enabling the visualization of viral particles and is widely used in research, diagnostics and morphological characterization of pathogens. Computer-assisted image analysis can help facilitate and speed up this labor-intensive method, however, typical EM virus datasets are small and sparsely labeled, resulting in a lack of readily available deep learning models due to training limitations. To overcome data shortages and leverage content-rich, diverse sample backgrounds, we employed background-based data augmentation. To this end, we propose Bg-Mix, a form of data augmentation method, which decomposes training images into foreground objects and backgrounds and subsequently rearranges both into novel augmented images. This not only enhances the performance of object detection models but also increases the diversity of the training data. Using Bg-Mix provides an increase between +0.8 and +3.3 box mean average precision (mAP) points over the baseline when tested across two datasets using two architectures, thereby demonstrating its effectiveness.
BibTeX
@misc{srivastava2026bg,
  author = {Ankur Srivastava and Joana Sarah Grah and Katharina Ladewig and Nils Körber},
  title = {{Bg-Mix: On Using Image Backgrounds for Data Augmentation in Electron Microscopy Images}},
  year = {2026},
  eprint = {2611.03455},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.03455},
}
Figure