Scope2Screen: Focus+Context Techniques for Pathology Tumor Assessment in Multivariate Image Data

Jared Jessup, Robert Krüger, Simon Warchol, John Hoffer, Jeremy Muhlich, Cecily C. Ritch, Giorgio Gaglia, Shannon Coy, Yu-An Chen, Jia-Ren Lin, Sandro Santagata, Peter K. Sorger, Hanspeter Pfister

TL;DR

Scope2Screen is a scalable visualization system designed to help pathologists explore massive, "high-plex" tissue images that contain up to 60 channels of molecular data. By using interactive "lenses" that overlay single-cell statistics and alternative channel views onto the global tissue context, it allows researchers to bridge the gap between traditional microscopy and modern multivariate data analysis.

What Problem Does This Paper Address?

Traditional pathology relies on H&E staining, which uses two colors to show tissue structure. Modern "high-plex" imaging techniques (like CyCIF or CODEX) can capture 60+ molecular markers simultaneously, producing 100GB+ images with millions of individual cells.

This creates a massive visualization challenge:

  1. Information Overload: Humans cannot perceive 60 colors at once; typical displays are limited to RGB.
  2. Loss of Context: When pathologists zoom in to see single-cell details, they often lose track of where they are in the larger tumor environment.
  3. Workflow Gaps: Switching between different channel combinations often leads to "change blindness," where the observer misses subtle differences because the entire screen updates at once.

Methods

Scope2Screen introduces a "scope-to-screen" metaphor, using virtual lenses to provide Focus+Context exploration.

  • Interactive Lensing: Users can move a virtual lens over the image. While the background (context) might show basic tissue morphology, the lens (focus) can show a completely different set of immune markers or high-resolution details.
  • Feature Augmentation: The lens acts as a data probe. As it moves, the system performs real-time spatial queries on an underlying CSV of single-cell data to display binned histograms or radial charts of marker intensities for just the cells under the lens.
  • HistoSearch (Search-by-Example): To navigate 100GB images, the authors implemented a sliding-window search. A user can place the lens over a region of interest (e.g., a specific immune cluster), and the system scans the entire image to find regions with similar intensity distributions using Chi-square distance.
  • The "Dotter" System: Inspired by pathologists physically "dotting" glass slides, this panel stores "rich snapshots." These aren't just images; they save the exact channel configurations, color settings, and single-cell statistics, allowing researchers to instantly restore a specific analytical state.
The Scope2Screen interface showing (A) channel selection, (B) the main viewer, (C) interactive lensing with histograms, and (D) the Dotter panel for snapshots.

Datasets

The system was evaluated using two large-scale cancer datasets:

  1. Colorectal Cancer (CRC1): A 40-channel CyCIF image containing 1.28 million cells (87.66 GB).
  2. Lung Adenocarcinoma: A 38-channel CyCIF image containing over 534,000 cells (118.43 GB). Both datasets included adjacent sections stained with traditional H&E for morphological comparison.

Key Insights and Ideas

  • Morphology is Sacred: During user studies, pathologists rejected "fisheye" lenses. While fisheye lenses are common in visualization research, they distort the shapes of nuclei and cells—the very features pathologists use for diagnosis. The authors pivoted to a "Plateau" lens that maintains a 1:1 scale in the center and only compresses the periphery.
  • The Power of Juxtaposition: The "split-screen" lens (showing H&E on one side and CyCIF on the other) was one of the most praised features. It allowed experts to validate that molecular markers aligned perfectly with known physical structures.
  • Search as Navigation: In images with billions of pixels, manual panning is inefficient. Using the HistoSearch to find "more like this" transformed the workflow from a random walk to a targeted investigation.
The HistoSearch workflow: (1) extracting a histogram from the lens, (2) sliding a window over the image, (3) computing similarity, and (4) overlaying results.

Key Results

  • Real-time Performance: The WebGL-based renderer handles 100GB+ images smoothly by leveraging image pyramids and chunked data loading (Zarr/OME-TIFF).
  • Biological Discovery: In the lung cancer case study, researchers used Scope2Screen to identify and characterize 10–12 Tertiary Lymphoid Structures (TLS)—complex immune cell aggregates—much faster than manual annotation would allow.
  • Validation: Domain experts (pathologists and biologists) rated the interface as highly intuitive, specifically noting that the proximity of single-cell stats to the visual image helped them "check their biases" when identifying cell types.

Limitations and Future Work

  • 2D Constraint: The current system is optimized for 2D whole-slide images. The authors acknowledge that high-plex 3D tissue data is emerging and will require new visual metaphors.
  • Spatial Analytics: While the tool shows distributions, it lacks advanced spatial statistics (e.g., calculating the exact distance between every T-cell and the nearest tumor cell) within the interactive loop.
  • Cloud Integration: Currently a local/server-based web app, future versions aim to support multi-dataset analysis across cloud-based "atlases" like the Human Tumor Atlas Network.

Summary updated

Abstract

Inspection of tissues using a light microscope is the primary method of diagnosing many diseases, notably cancer. Highly multiplexed tissue imaging builds on this foundation, enabling the collection of up to 60 channels of molecular information plus cell and tissue morphology using antibody staining. This provides unique insight into disease biology and promises to help with the design of patient-specific therapies. However, a substantial gap remains with respect to visualizing the resulting multivariate image data and effectively supporting pathology workflows in digital environments on screen. We, therefore, developed Scope2Screen, a scalable software system for focus+context exploration and annotation of whole-slide, high-plex, tissue images. Our approach scales to analyzing 100GB images of 109 or more pixels per channel, containing millions of individual cells. A multidisciplinary team of visualization experts, microscopists, and pathologists identified key image exploration and annotation tasks involving finding, magnifying, quantifying, and organizing regions of interest (ROIs) in an intuitive and cohesive manner. Building on a scope-to-screen metaphor, we present interactive lensing techniques that operate at single-cell and tissue levels. Lenses are equipped with task-specific functionality and descriptive statistics, making it possible to analyze image features, cell types, and spatial arrangements (neighborhoods) across image channels and scales. A fast sliding-window search guides users to regions similar to those under the lens; these regions can be analyzed and considered either separately or as part of a larger image collection. A novel snapshot method enables linked lens configurations and image statistics to be saved, restored, and shared with these regions. We validate our designs with domain experts and apply Scope2Screen in two case studies involving lung and colorectal cancers to discover cancer-relevant image features.
BibTeX
@article{jessup2022scope,
  author = {Jared Jessup and Robert Krüger and Simon Warchol and John Hoffer and Jeremy Muhlich and Cecily C. Ritch and Giorgio Gaglia and Shannon Coy and Yu-An Chen and Jia-Ren Lin and Sandro Santagata and Peter K. Sorger and Hanspeter Pfister},
  title = {{Scope2Screen: Focus+Context Techniques for Pathology Tumor Assessment in Multivariate Image Data}},
  journal = {IEEE Trans. Vis. Comput. Graph.},
  year = {2022},
  url = {https://openreview.net/forum?id=PSoj1GcJtQ},
}

Loading related content...

Figure