Tell me what happened here in history

Jia Chen, Qin Jin, Weipeng Zhang, Shenghua Bao, Zhong Su, Yong YuACM Multimedia 2013 · DOI

TL;DR

This paper introduces LMHistory, a system that allows users to query historical landmarks with an image and receive a chronological timeline of related events and photos. It addresses the gap in landmark research by focusing on the temporal dimension, offering a novel way to explore the history behind places.

What Problem Does This Paper Address?

In an era of ubiquitous smartphones and increasing interest in travel, people often find themselves curious about the history of landmarks they encounter. Existing landmark recognition systems primarily focus on identifying the landmark itself or its spatial context. However, there's a lack of systems that effectively connect a landmark image to its rich historical narrative. This paper aims to bridge this gap by developing a system that not only recognizes a landmark from an image but also presents its historical events in a chronological and visually engaging manner, enriching the user's travel experience.

Methods

The LMHistory system is built upon three core interfaces: recognition, history summary, and history detail.

  • Recognition Interface: This interface uses an image indexing module that leverages SURF features for image retrieval. Landmarks are ranked based on the top 100 retrieved images. The system can process image queries via URL or direct upload, providing landmark recognition in under a second.
  • Historical Event and Photo Mining Module: This module is the backbone for the history summary and detail interfaces. It mines historical events from Wikipedia, representing each event with a triplet of location, time, and characters. To enrich these events, the system queries multiple online encyclopedias and image search engines (Flickr, Wikipedia, Google, Bing, Yahoo) to gather related photos.
  • Event and Photo Quality Scoring: To ensure the relevance and quality of mined information, the system employs scoring mechanisms. A score is assigned to each retrieved photo based on its rank in search engine results, content similarity to other images, and the associated webpage. Additionally, a separate score evaluates the overall quality of the image collection for each event.

Datasets

The paper mentions the following datasets and resources:

  • Landmark Recognition: 106 landmarks and approximately 79,773 photos are indexed for recognition.
  • Historical Event and Photo Mining: Data is mined from Wikipedia for historical events. Photos are retrieved from multiple online image search engines, including Flickr, Google, Bing, and Yahoo.

Key Insights and Ideas

  • Temporal Dimension is Key: The most significant insight is the focus on the temporal dimension of landmarks, which has been largely unexplored in prior research. This shifts the focus from "what is this place?" to "what happened here over time?"
  • Bridging Online Trends: The system effectively combines two growing trends: image-based landmark recognition and the desire for instant historical information on mobile devices.
  • Multi-Source Data Fusion: The system demonstrates the power of fusing information from diverse online sources (Wikipedia, image search engines) to create a comprehensive historical narrative.
  • Automated Quality Control: The development of scoring mechanisms for both individual photos and event collections is crucial for filtering out irrelevant or low-quality information from the vastness of the internet.

Key Results

  • Efficient Recognition: The landmark recognition process takes less than a second.
  • Comprehensive Data: The system has mined a total of 612 events for 106 landmarks, accompanied by 5,497 historical photos.
  • User-Friendly Interface: The system provides a convenient interface for users to search by image, browse events chronologically, and explore historical photos in detail.

Limitations and Future Work

  • Limitations: The paper does not explicitly detail limitations in the provided text. However, a potential limitation could be the reliance on the completeness and accuracy of information available on Wikipedia and the quality of results from image search engines. The confidence score for events is mentioned, implying that not all mined information is perfectly reliable.
  • Future Work: The authors suggest wider application scenarios such as compiling history data for online museums and illustrating history in education. While not explicitly stated as future work, the current system's reliance on specific online sources could be expanded to include more diverse data repositories.

Summary updated

BibTeX
@inproceedings{chen2013tell,
  author = {Jia Chen and Qin Jin and Weipeng Zhang and Shenghua Bao and Zhong Su and Yong Yu},
  title = {{Tell me what happened here in history}},
  booktitle = {ACM Multimedia},
  year = {2013},
  doi = {10.1145/2502081.2502272},
}
Figure