SILA: a system for scientific image analysis

Daniel Moreira, João Phillipe Cardenuto, Ruiting Shao, Sriram Baireddy, Davide Cozzolino, Diego Gragnaniello, Wael AbdAlmageed, Paolo Bestagini, Stefano Tubaro, Anderson Rocha, Walter J. Scheirer, Luisa Verdoliva, Edward J. DelpScientific 2022 · DOI

Abstract

A great deal of the images found in scientific publications are retouched, reused, or composed to enhance the quality of the presentation. In most instances, these edits are benign and help the reader better understand the material in a paper. However, some edits are instances of scientific misconduct and undermine the integrity of the presented research. Determining the legitimacy of edits made to scientific images is an open problem that no current technology can perform satisfactorily in a fully automated fashion. It thus remains up to human experts to inspect images as part of the peer-review process. Nonetheless, image analysis technologies promise to become helpful to experts to perform such an essential yet arduous task. Therefore, we introduce SILA, a system that makes image analysis tools available to reviewers and editors in a principled way. Further, SILA is the first human-in-the-loop end-to-end system that starts by processing article PDF files, performs image manipulation detection on the automatically extracted figures, and ends with image provenance graphs expressing the relationships between the images in question, to explain potential problems. To assess its efficacy, we introduce a dataset of scientific papers from around the globe containing annotated image manipulations and inadvertent reuse, which can serve as a benchmark for the problem at hand. Qualitative and quantitative results of the system are described using this dataset.

Introduction: The Challenge of Image Integrity

In the contemporary scientific landscape, images are not merely illustrative; they constitute primary data. However, the integrity of this data is increasingly under threat. Between 2011 and 2015, the US Office of Research Integrity found that nearly 78% of research misconduct cases involved some form of image manipulation. While many edits—such as contrast adjustments—are benign attempts to improve clarity, others involve deceptive practices like "copy-move" cloning, "splicing" from different sources, or "repurposing" data across unrelated experiments.

A modified scientific image demonstrating common retouching or manipulation techniques.

The fundamental challenge in addressing this issue is that fully automated detection remains a "holy grail." Benign and malicious edits often appear identical to algorithms, and the sheer volume of publications makes manual inspection by peer reviewers an arduous and error-prone task. We introduce SILA (Scientific Image Analysis), a "human-in-the-loop" end-to-end system designed to augment human expertise by providing principled evidence of potential misconduct.

The SILA Framework

SILA implements a five-stage computational workflow to process and analyze scientific figures:

  • Content Extraction: Utilizing tools like MuPDF and PDFMiner, the system extracts images and their associated captions directly from the PDF binary stream.
  • Panel Segmentation: Scientific figures are frequently "multiplexed," containing multiple sub-panels (e.g., A, B, C). SILA employs a YOLOv2 convolutional neural network to automatically segment these into individual units for granular analysis.
  • Image Ranking: To detect reuses across disparate publications, the system uses SURF interest points and RootSIFT descriptors to build a searchable index, allowing analysts to identify similar content across massive databases.
  • Copy-Move Detection: This module is tailored for the scientific domain by fusing Zernike moments (robust to rotation) with RGB channel data (robust to intensity changes). It also incorporates OCR to remove text overlays, which are common sources of false positives.
  • Provenance Analysis: The system generates Provenance Graphs—Directed Acyclic Graphs (DAGs) that visualize the "family tree" of an image. These graphs trace how content has been cropped, rotated, or spliced across multiple papers over time.
Example of image splicing where a cell from a donor image is inserted into a host image, illustrating the donor-host relationship tracked by SILA's provenance graphs.

The Scientific Papers (SP) Dataset

To establish a benchmark for the community, we introduced the Scientific Papers (SP) dataset. This collection includes 988 publications, featuring 298 retracted papers with documented image issues and 332 "clean" biomedical papers from PubMed Central. The dataset provides ground-truth annotations for 1,876 figures, including 180 pixel-wise copy-move masks and 70 unique provenance graphs.

Results and Discussion

Evaluation on the SP dataset demonstrated the system's efficacy. SILA achieved an image extraction recall of 71%. In the challenging task of copy-move detection, the system reached an F1-score of 0.35—a state-of-the-art result for scientific imagery where cloned regions are often subtle and low-contrast. Provenance analysis achieved a Vertex Overlap (VO) of 0.64, indicating high accuracy in identifying related images, though reconstructing the exact sequence of edits (Edge Overlap of 0.16) remains more difficult.

A key technical insight was the necessity of feature fusion. While Zernike features perform well on western blots, they are less effective for cell microscopy; conversely, RGB features excel in microscopy but struggle with blots. Fusing these features created a significantly more robust detector. Additionally, using publication dates as a constraint in provenance graphs ensures the "arrow of time" is respected, reducing search complexity.

Conclusion

SILA represents a significant step toward safeguarding scientific integrity. By focusing on providing evidence—such as masks and provenance graphs—rather than automated verdicts, the system respects the role of human judgment in determining intent. Future work will address emerging threats like AI-generated forgeries and sophisticated content removal techniques.

Summary updated

Abstract

A great deal of the images found in scientific publications are retouched, reused, or composed to enhance the quality of the presentation. In most instances, these edits are benign and help the reader better understand the material in a paper. However, some edits are instances of scientific misconduct and undermine the integrity of the presented research. Determining the legitimacy of edits made to scientific images is an open problem that no current technology can perform satisfactorily in a fully automated fashion. It thus remains up to human experts to inspect images as part of the peer-review process. Nonetheless, image analysis technologies promise to become helpful to experts to perform such an essential yet arduous task. Therefore, we introduce SILA, a system that makes image analysis tools available to reviewers and editors in a principled way. Further, SILA is the first human-in-the-loop end-to-end system that starts by processing article PDF files, performs image manipulation detection on the automatically extracted figures, and ends with image provenance graphs expressing the relationships between the images in question, to explain potential problems. To assess its efficacy, we introduce a dataset of scientific papers from around the globe containing annotated image manipulations and inadvertent reuse, which can serve as a benchmark for the problem at hand. Qualitative and quantitative results of the system are described using this dataset.
BibTeX
@inproceedings{moreira2022sila,
  author = {Daniel Moreira and João Phillipe Cardenuto and Ruiting Shao and Sriram Baireddy and Davide Cozzolino and Diego Gragnaniello and Wael AbdAlmageed and Paolo Bestagini and Stefano Tubaro and Anderson Rocha and Walter J. Scheirer and Luisa Verdoliva and Edward J. Delp},
  title = {{SILA: a system for scientific image analysis}},
  booktitle = {Scientific},
  year = {2022},
  doi = {10.1038/s41598-022-21535-3},
}
Figure