TL;DR
This paper introduces "WonderWhat," a system that identifies public events in real-time by analyzing photos taken by users. It leverages spatio-temporal context and image content to provide users with relevant event information, addressing the challenge of discovering local activities while traveling.
What Problem Does This Paper Address?
Navigating unfamiliar environments, especially when traveling, often makes it difficult for people to discover and participate in authentic local events. Traditional search engines rely heavily on keyword matching, which can be insufficient for capturing the nuances of event discovery. Existing semantic web and visual search approaches, while advanced, still struggle with the rich semantic meaning of "events" and the "semantic gap" between user intent and available data. The authors aim to bridge this gap by creating a system that can automatically determine the event a user is attending based on a photo they take.
Methods
WonderWhat employs a multi-step approach to determine events from user photos:
- Event Database Ingestion: The system maintains a dedicated database of both planned events (extracted from web pages and services) and emergent events (detected from real-time micro-blogs like Twitter).
- Contextual Photo Query: When a user takes a photo, the system receives it along with its associated time and location data (forming a "contextual photo").
- Spatio-Temporal Filtering: The initial query uses the photo's time and location to retrieve a list of potentially relevant events from the database.
- Content-Based Analysis: A crucial component analyzes the image content to infer the event type. This involves a learning-based approach that first trains concept detectors, then uses these to detect concepts within event-associated photos, and finally trains an event type detector. This allows the system to understand visual characteristics of different event types.
- Ranking: A ranking component combines the results from the event database (spatio-temporal match) and the content analysis (visual match). It considers spatial, temporal, and visual distances to produce a final ranked list of events.
Datasets
- Flickr Dataset: Used to verify the hypothesis that people take photos at events and to test the system's ability to match photos to events using time and location. The event database for this experiment was built for NYC events from 2008-2010, and photos were downloaded via the Flickr API for the same period.
- Real Event Photo Set: A custom dataset collected from four volunteers in NYC during August and September 2010. Volunteers were instructed to take photos at events they discovered, providing a more direct evaluation of the system's real-world performance.
Key Insights and Ideas
- Bridging the Semantic Gap for Events: The paper highlights that events are complex concepts with rich semantic meaning that traditional keyword or object-based visual search struggles to capture. WonderWhat's approach of combining spatio-temporal context with visual content is key to addressing this.
- Leveraging User-Generated Content: By using photos taken by users, the system taps into a rich source of real-time, contextual information that is directly relevant to the user's experience.
- The Power of Combined Context: The effectiveness of the system hinges on the synergistic use of multiple context sources: when the photo was taken, where it was taken, and what is visually depicted in the photo.
- Distinguishing Event Types Visually: The insight that different event types have distinct visual characteristics is fundamental to the content-based analysis component. This allows for a more nuanced understanding beyond just location and time.
Key Results
- The system successfully matched photos to corresponding events in the Flickr dataset, demonstrating the feasibility of using spatio-temporal information for event discovery.
- Experiments on the real photo set showed promising results, with the system correctly identifying the corresponding event in the first place of the ranked list for most photos.
- For one case where the exact event wasn't in the database, the system returned a "very close match" (a musical event when the ground truth was a festival), indicating robustness.
Limitations and Future Work
- Database Completeness: The paper acknowledges that the accuracy of event determination is dependent on the completeness and accuracy of the event database. For instance, in one experiment, the system returned a close match because the exact event was not stored.
- Emergent Event Detection: While the system aims to detect emergent events from micro-blogs, the paper doesn't detail the performance or challenges associated with this specific component.
- Future Work: The authors suggest that further work could focus on improving the event database and refining the content analysis and ranking components. The potential for discovering concurrent events by analyzing real-time micro-blogs is also highlighted as a promising direction.
Summary updated