Do Memorable Findings Hide Subtle Ones? A Retrospective Read-Cohort Study of Chest X-rays

When a radiologist finds one striking abnormality on a chest X-ray, they are more likely to miss a second, quieter one. Clinicians call this "satisfaction of search." It is usually explained with broad statements about human heuristics, and nobody has measured it on the image itself. A separate line of vision research offers a way to do that measurement. Granular memorability models score how likely each point or object in an image is to be noticed and retained by a human viewer. That work has found that a scene tends to be represented by its single most memorable object, the "peak object." We propose to join the two. We will score each chest radiograph for how strongly one region dominates its point-level memorability, and then use the logic of a retrospective cohort study to test whether that dominance predicts missed secondary findings. A retrospective cohort study analyzes records that already exist, defines an exposure, follows the outcome forward, and adjusts statistically for confounders. Here the records are archived reading sessions.

The core claim is specific and can be falsified. Radiologists miss secondary pathologies more often on cases where a single region has much higher point memorability than every other region. This holds after matching on case difficulty, number of findings, reader experience and AI assistance, and it holds within a case: readers whose logged interactions show early capture by the peak region miss more. The claim is only interesting if a memorability score predicts this better than a plain saliency map or a lesion-conspicuity proxy, and the study is built so that it can lose to those baselines.

Why this intersection, and what already exists

The memorability side supplies the measurement. Security Analyses of Click-based Graphical Passwords via Image Point Memorability defines Image Point Memorability (IPM) from three parts:

  • semantic coherence, meaning how meaningful the point's object is;
  • salience, meaning how much the object stands out globally;
  • distinguishability, meaning how much a point differs from its immediate neighborhood, which is the part intended to track memory decay.

That paper showed that human point choices cluster so tightly on memorable hotspots that a dictionary built from these scores broke 55.21% of Persuasive Cued Click Points passwords. Spatial Mapping of Visual Memorability reports the peak-object phenomenon. It also reports that localized memorability maps correlate only weakly with global image scores, and that saliency does not guarantee retention in cluttered scenes. AMNet: Memorability Estimation With Attention found that memorable images produce concentrated attention maps and forgettable ones produce diffuse maps. We borrow the measurement and the peak-object hypothesis from these papers. All three were validated only on natural images and lab or crowd-sourced cohorts, and none of them studies what a dominant anchor does to the search for everything else.

The data comes from A Dataset for Understanding Radiologist-Artificial Intelligence Collaboration, known as Collab-CXR. It contains 227 radiologists reading 324 historical Stanford chest X-rays across 104 pathologies. Each read is labeled against a 5-expert consensus and comes with click-stream, pan, zoom, contrast, active-time and AI-condition logs. That paper analyzes accuracy and AI collaboration but never models the image's memorability structure, and it is a controlled experiment rather than an observational cohort. We reuse it as a historical archive and analyze it retrospectively. The file shows no paper at this intersection.

The cohort side supplies the adjustment machinery. Two clinical studies set the template. Long-Term Survival in Men With High Grade Prostate Cancer: A Comparison Between Conservative Treatment, Radiation Therapy and Radical Prostatectomy—A Propensity Scoring Approach uses propensity score matching, which pairs units that had similar predicted probability of exposure, to separate treatment effects from confounding by indication. Risk factors associated with 31-day unplanned readmission in 50,912 discharged patients after stroke in China uses multilevel logistic models to absorb institution-level clustering. The analogous threat here is sharp: cases with a dominant, memorable lesion are probably also sicker and more complex. Both papers also carry a warning that constrains our design. Unmeasured confounding survives any matching, and so does missingness such as the prostate cohort's absent early comorbidity data. For that reason the within-case analysis below carries the main weight, and the between-case analysis is treated as supporting evidence.

Research Program

The first step is a memorability scorer for radiographs. Natural-image models trained on the Large-scale Memorability dataset (LaMem) do not transfer to grayscale X-rays, so we rebuild IPM for this domain:

  • Distinguishability is computed from local gradient and intensity dissimilarity, as in the automated IPM pipeline.
  • Salience comes from a generic visual attention model.
  • Semantic coherence is assigned from anatomy and pathology labels, splitting findings into high-, medium- and low-coherence classes by clinical meaningfulness.

Before any outcome analysis, we will check that this score predicts where radiologists actually go. On held-out readers, the score should correlate with click and zoom density. This mirrors how Spatial Mapping of Visual Memorability built Visual Memory Schemas from pooled human annotations. If the score fails this check, the study stops there and reports the negative transfer result.

From each case's point memorability map we compute a peak-dominance ratio: the mean memorability of the top region divided by that of the next-highest region. This is the case-level exposure. The outcome is a missed secondary finding: a consensus-positive pathology that is not the case's highest-memorability region and that the reader failed to report. As a negative-control outcome we use detection of the peak finding itself. Peak dominance should not harm, and may help, detection of the peak.

We then run two analyses.

The first is between cases. We propensity-match high- versus low-dominance cases on number of consensus findings, pathology categories, lesion size proxies, patient history availability and AI condition. We fit a multilevel logistic model with reader random effects, in the style of the stroke readmission study.

The second is within cases and is the stronger design. It uses a reader-level exposure taken from the logs: whether the reader's first sustained zoom or click cluster landed on the peak region, and what share of active time they spent there. We include case fixed effects, which remove every fixed property of the image, including its difficulty. We adjust for reader experience and AI condition. Because readers are not randomized to where they look, we also check how much of the within-case association is explained by reader skill. We refit with reader fixed effects on the subset of readers who contributed enough cases.

We will be explicit about sample size. The between-case analysis rests on only 324 images, so it can detect only fairly large effects, and we will report detectable-effect sizes in advance. The within-case analysis draws on many reads per case and is where most of the statistical power lies.

Baselines

The naive reference is a model with no image information. It predicts misses from each reader's base miss rate, the number of consensus findings, and AI condition. This is the "just count abnormalities" account of satisfaction of search. The memorability exposure must improve held-out log-loss and area under the ROC curve over it.

The strongest competitor is a saliency-only version of the same pipeline. We compute peak dominance from a generic visual attention map, which is the salience component alone, and run the identical matching, the identical multilevel and fixed-effect models, and the identical reader splits. Saliency maps are cheap and are what a reviewer would reach for first. Beating this baseline requires a significantly larger and better-calibrated miss association for full-IPM dominance on held-out readers. If the two are statistically indistinguishable, we will report that the effect is attentional capture rather than memorability.

A second strong competitor is a global memorability score in the style of Understanding and Predicting Image Memorability at a Large Scale. This is MemNet, which reached a Spearman rank correlation of 0.64 on LaMem against a human ceiling of 0.68. It is applied per case and entered as a covariate. It tests whether a single image-level number already captures the effect, which would undercut the granular argument. AMNet's 0.677 Spearman correlation on LaMem is the best reported memorability number in the file. It is a natural-image figure, though, and says nothing about radiology. For the actual question, whether memorability predicts radiologist misses, I found no reported result on Lacuna.

Ablations remove distinguishability and semantic coherence one at a time. They show which IPM ingredient carries any effect. If semantic coherence alone explains it, the finding reduces to "clinically meaningful findings distract," which is still useful but is a different claim.

Why not just…?

Why not just use a saliency map? This is the most serious threat. Satisfaction of search may simply be attention capture by a salient lesion, with no role for memory encoding. The memorability papers give some reason to expect a difference. Spatial Mapping of Visual Memorability reports that saliency does not guarantee retention in clutter, and IPM's distinguishability term measures contrast with the immediate neighborhood rather than global pop-out. None of that evidence comes from radiographs. The saliency-only baseline, run under identical conditions, settles the question, and we will accept a null.

Why not just count the findings and adjust for difficulty? More findings mechanically create more chances to miss one, and hard cases are both memorable and error-prone. This is exactly why the within-case fixed-effect analysis exists. Every image-level property, finding count included, is held constant, and only reader-specific capture by the peak region varies. If the effect disappears under case fixed effects, it was confounding by case.

Why not run a prospective randomized experiment instead? An experiment could randomize readers to images with a dominant anchor digitally attenuated. That would give a cleaner causal answer, and it is the right follow-up. It would cost new radiologist time, and it would manipulate images away from real clinical appearance. Evaluating Clinical Outcomes Using Historical Patient Cohorts argues that retrospective designs are the cheap first filter and that prospective confirmation should follow, as Mitigating Retrospective Bias insists. We position this study as that filter. It decides whether an anchor-attenuation trial is worth running.

Decisive Checks

The idea survives only if four things hold:

  • The radiograph memorability score predicts held-out reader click density better than chance.
  • Peak dominance predicts secondary misses under case fixed effects.
  • The association is larger than the saliency-only baseline's.
  • The negative-control outcome, detection of the peak finding itself, shows no harm.

A clean failure at the saliency comparison is still informative. It would say that the "memorable anchor" story in radiology is really about attention capture, and that granular memorability adds nothing there. That tells the memorability community where its measures stop generalizing.