PhotoScout: Synthesis-Powered Multi-Modal Image Search

Celeste Barnaby, Qiaochu Chen, Chenglong Wang, Isil Dillig

Introduction

Modern photo management systems excel at keyword-based searches like "dog" or "beach." However, they frequently falter when faced with structured image retrieval—tasks defined by complex semantic relationships and logical constraints. For instance, a user might seek a specific moment where "the bride is to the left of the groom while holding flowers." Such queries expose the "semantic gap" in current vision models, which struggle to reason about high-level spatial and logical relationships.

A complex group shot requiring logical reasoning about spatial relationships between subjects.

System Architecture: Neuro-Symbolic Synthesis

PhotoScout addresses these challenges through a neuro-symbolic program synthesis approach. Rather than relying solely on embedding similarity, the system constructs a formal program to execute the search.

  1. Natural Language to Sketch: An LLM (GPT-3.5 Turbo) translates natural language queries into "program sketches" within a custom Domain-Specific Language (DSL) based on first-order logic.
  2. Grounding and Interaction: To resolve ambiguities (e.g., identifying specific individuals), users provide visual examples or tag objects. These "ground" the abstract logic in visual data.
  3. Program Synthesis: PhotoScout performs an enumerative search to fill "holes" in the program sketch, identifying the simplest logical structure that satisfies all user-provided positive and negative examples.
  4. Execution: The final program leverages neural models (Amazon Rekognition) for object detection and attribute recognition, then applies the synthesized logic across the image library.

Evaluation and Results

The system was evaluated using three distinct Flickr-sourced datasets: Transportation, Festival, and Wedding. These datasets represent personal collections where structured queries are most prevalent.

Sample images from the Wedding dataset used to evaluate PhotoScout's retrieval capabilities.

In a user study with 25 participants, PhotoScout was compared against a CLIP-based baseline (ClipWrapper). Key findings include:

  • Improved Accuracy: PhotoScout achieved a 34% higher F1 score than the baseline.
  • Efficiency and Confidence: While synthesis introduces a brief computational delay (up to 4.8 seconds), users reported higher confidence in results and required fewer manual refinements.
  • Explainability: The symbolic nature of the programs allows the system to provide natural language explanations for its selections, fostering user trust.

Discussion and Future Work

While PhotoScout significantly advances structured retrieval, it remains dependent on the accuracy of underlying object detectors. Failures in detection (e.g., obscured faces) propagate to the logical layer. Future research will explore "human-in-the-loop" corrections, allowing users to refine detection results to further improve synthesis outcomes.

Summary updated

BibTeX
@inproceedings{barnaby2024photoscout,
  author = {Celeste Barnaby and Qiaochu Chen and Chenglong Wang and Isil Dillig},
  title = {{PhotoScout: Synthesis-Powered Multi-Modal Image Search}},
  booktitle = {CHI},
  year = {2024},
  doi = {10.1145/3613904.3642319},
}
Figure