Abstract
The rise of generative models has led to increased use of large-scale datasets collected from the internet, often with minimal or no data curation. This raises concerns about the inclusion of sensitive or private information. In this work, we explore the presence of pregnancy ultrasound images, which contain sensitive personal information and are often shared online. Through a systematic examination of LAION-400M dataset using CLIP embedding similarity, we retrieve images containing pregnancy ultrasound and detect thousands of entities of private information such as names and locations. Our findings reveal that multiple images have high-risk information that could enable re-identification or impersonation. We conclude with recommended practices for dataset curation, data privacy, and ethical use of public image datasets.
Introduction: The Privacy Blind Spot in Large-Scale Scraping
The current paradigm for building massive image datasets, such as LAION, often follows a "scrape first, ask questions later" philosophy. While audits have historically focused on NSFW content or societal biases, a critical category of sensitive data has remained largely unexamined: pregnancy ultrasounds.
Expectant parents frequently share sonograms on social media to celebrate milestones. However, when these images are ingested into AI training sets, they lose their original context. Crucially, these medical images often contain "burned-in" metadata—patient names, hospital locations, birth dates, and sonographer IDs. This practice constitutes a breach of contextual integrity, where data intended for a specific social circle is repurposed for public research without consent, creating risks for re-identification or impersonation.
Methodology: Detection and PII Extraction
To audit the LAION-400M dataset, we developed a two-stage pipeline designed to identify ultrasounds and extract sensitive personally identifiable information (PII).
1. Detection Strategies
We compared two primary methods for locating ultrasounds within the 400 million images:
- Retrieval-based: Utilizing CLIP embeddings to find images similar to specific text queries (e.g., "prenatal ultrasound") or reference images.
- Classifier-based: Training dedicated machine learning models (SVM, Random Forest, and Neural Networks) on top of CLIP features to distinguish between positive ultrasound images and complex negatives.
2. PII Extraction Pipeline
Once identified, images underwent a sophisticated extraction process involving preprocessing with Real-ESRGAN for upscaling and rotation. Initial text recognition via Tesseract was refined using LLaVa-Next, a vision-language model that corrects OCR errors by analyzing the visual context. Finally, Microsoft Presidio was used to categorize the text into PII types such as Name, Location, and Date.
Datasets and Key Results
Our audit targeted the LAION-400M dataset. To train our classifiers, we introduced the PIU (Pregnancy Image Ultrasound) Dataset, containing 9,900 balanced images, including "hard negatives" like ultrasound machines or baby shower decor.
Detection Performance: The SVM classifier proved most effective, achieving 97.27% accuracy on the PIU test set with a minimal false-positive rate of 1.41%.
Audit Findings: * 833 unique pregnancy ultrasound images were identified within LAION-400M. * 677 instances of PII were extracted from these images, including 387 Names and 238 Locations. * High-Risk Overlap: Approximately 10% of identified images contained multiple PII types, significantly increasing re-identification risks.
Insights: The Diversity of Ultrasound Sharing
A key finding is that ultrasounds in public datasets are not limited to clinical films. They appear in diverse contexts, such as baby shower invitations, pregnancy announcements, and sentimental keepsakes. This variety underscores why simple keyword filtering is insufficient for dataset curation and highlights the complexity of managing contextual integrity.
Conclusion and Future Directions
The presence of sensitive medical data in public AI datasets highlights a failure in maintaining contextual integrity. While OCR on low-resolution medical films remains challenging, the use of LLMs as "OCR fixers" demonstrates a powerful path forward for auditing. We recommend that future dataset curation move toward informed consent models and implement differential privacy to ensure that models do not inadvertently memorize and regurgitate private medical information.
Summary updated