AEye: A Visualization Tool for Image Datasets

Florian Grötschla, Luca A. Lanzendörfer, Marco Calzavara, Roger Wattenhofer

Abstract

AEye is an open-source visualization tool designed to transform massive, "black box" image datasets into intuitive, navigable 2D maps. By combining CLIP semantic embeddings with a novel hierarchical tiling system, it allows researchers to explore datasets from a high-level overview down to individual images while maintaining semantic context and visual continuity.

Introduction

In modern artificial intelligence, the quality and composition of training data dictate model performance and inherent biases. However, as datasets scale to hundreds of thousands or millions of images, they become nearly impossible to audit manually. Traditional methods, such as scrolling through grids or using standard dimensionality reduction projectors (like t-SNE or UMAP), often fail at scale. Grids lack semantic organization, and standard projectors frequently result in a cluttered "hairball" that overwhelms both the user's cognitive load and the browser's rendering capabilities. AEye addresses these challenges by providing a structured exploration tool that allows users to move seamlessly between a "bird's-eye view" and a "worm's-eye view."

Methods

The AEye pipeline consists of four primary stages designed to organize and present data semantically:

  1. Semantic Embedding: Every image is processed through a pretrained CLIP (Contrastive Language-Image Pretraining) model, converting images into 512-dimensional vectors where semantically similar images are mathematically adjacent.
  2. Dimensionality Reduction: The system uses UMAP to project the 512D space onto a 2D plane. UMAP is preferred over t-SNE for its ability to preserve global data structures while remaining scalable.
  3. Hierarchical Tiling & Clustering: This core innovation organizes the 2D plane into a grid of tiles across multiple layers. Using a modified k-means clustering algorithm, the system selects representative images for each tile. Crucially, as a user zooms in, representatives from higher layers are kept as fixed centroids, ensuring visual stability so "landmark" images do not disappear or shift unexpectedly.
  4. Interactive Interface: The web-based front end supports panning, zooming, and semantic search powered by a Milvus vector database. It also provides AI-generated captions (via LLaVA) to offer deeper context for individual images.
The AEye processing pipeline: from raw image dataset to interactive semantic map.

Hierarchical Tiling and Visual Continuity

A significant frustration in large-scale data visualization is "losing one's place" during navigation. AEye solves this by forcing the clustering algorithm to retain representatives from the parent layer as the user zooms. This creates a stable mental map, allowing for efficient exploration of 100k+ images on standard hardware without performance lag.

Hierarchical tiling and clustering layers ensure visual stability during zoom operations.

Evaluation and Results

The authors demonstrated AEye's utility across several benchmarks:

  • MNIST: Successfully clustered handwritten digits by visual and semantic similarity.
  • CelebA-HQ: Revealed high-level attribute distributions, such as the clear separation between demographic features in the embedding space.
  • COCO 2017: In a case study of 163,000 images, AEye helped practitioners identify labeling errors and underrepresented categories that would be impossible to find via random sampling.

Key results indicate that the tool makes dataset bias visually undeniable. For example, if a search for a specific profession only populates one corner of the map dominated by a single demographic, the bias is immediately apparent to the researcher.

Limitations and Future Work

While effective, AEye is subject to the inherent artifacts of dimensionality reduction; users must be cautious not to over-interpret mathematical gaps in the UMAP projection. Currently, the tool is optimized for static datasets; adapting the hierarchical clustering for real-time "streaming" data remains a challenge. Additionally, scaling to tens of millions of images (such as the LAION dataset) will require more optimized distributed computing for the preprocessing stages.

Conclusion

AEye provides a necessary bridge between raw image data and human-understandable insights. By prioritizing visual continuity and semantic organization, it empowers researchers to audit their data for quality and fairness more effectively than traditional grid-based or projection-based tools.

Summary updated

Abstract

Image datasets serve as the foundation for machine learning models in computer vision, significantly influencing model capabilities, performance, and biases alongside architectural considerations. Therefore, understanding the composition and distribution of these datasets has become increasingly crucial. To address the need for intuitive exploration of these datasets, we propose AEye, an extensible and scalable visualization tool tailored to image datasets. AEye utilizes a contrastively trained model to embed images into semantically meaningful high-dimensional representations, facilitating data clustering and organization. To visualize the high-dimensional representations, we project them onto a two-dimensional plane and arrange images in layers so users can seamlessly navigate and explore them interactively. AEye facilitates semantic search functionalities for both text and image queries, enabling users to search for content. We open-source the codebase for AEye, and provide a simple configuration to add datasets.
BibTeX
@inproceedings{grotschla2024aeye,
  author = {Florian Grötschla and Luca A. Lanzendörfer and Marco Calzavara and Roger Wattenhofer},
  title = {{AEye: A Visualization Tool for Image Datasets}},
  booktitle = {IEEE VIS},
  year = {2024},
  url = {https://openreview.net/forum?id=A6psiYpaTM},
}

Loading related content...

Figure