ObservatóR!O2016

Julia Giannella, Luiz VelhoDH 2017

Abstract

ObservatóR!O2016 is a research platform designed to analyze the massive visual output of the Rio 2016 Olympics on Twitter. By leveraging transfer learning with the Inception-v3 model, the project automates the classification of specific visual themes, such as the Olympic torch relay, across 180,000 images. The platform integrates these findings into interactive visualizations that facilitate serendipitous discovery within large-scale datasets, moving beyond simple text-based search to explore the "visual culture" of global events.

Introduction

The challenge of "discoverability" in massive, unstructured social media datasets is significant. During the Rio 2016 Olympics, millions of images were shared, but identifying specific visual themes like the Olympic torch relay is difficult using text analysis alone. Many tweets mention the torch without a photo, while others post photos without using the word in the text. To truly understand the event's visual culture, researchers require methods to classify images based on actual content rather than just metadata.

The ObservatóR!O2016 dashboard featuring eight interactive visualizations for exploring the Olympic debate.

Methods

The project utilizes a combination of computer vision and interactive web design:

  • Transfer Learning: Rather than training a neural network from scratch, the authors used TensorFlow and the Inception-v3 model. They retrained the final layers using a small, manually labeled sample of just 100 images containing the Olympic torch.
  • Automated Classification: The retrained model processed the entire database to assign confidence scores, identifying images most likely to contain the torch.
  • Mosaic Visualization: To present results, the team built a mosaic where a target image is reconstructed using hundreds of smaller "tile images" identified as torches.
  • Interactive UI: The platform implemented OpenSeadragon, allowing users to zoom and pan through high-resolution image collections.

Datasets

The research utilized a Twitter stream of approximately 1 million tweets collected between April and August 2016. From this stream, 180,000 unique images were identified. The training set for the transfer learning task consisted of a custom-labeled set of only 100 images.

Key Insights and Results

The retrained network successfully identified approximately 1,500 images with a confidence score over 85% for the "Olympic torch" category. A key insight is the efficiency of transfer learning; high-confidence classification was achieved with a remarkably small training set, making deep learning more accessible for humanities research.

A grid of identified torch images, including a spoof of the character Fofão, illustrating serendipitous discovery.

Furthermore, the project demonstrates the value of "serendipitous discovery." While the mosaic visualization has aesthetic appeal, it serves a functional purpose: zooming into the collection allows for surprises, such as discovering a spoof image of the Brazilian character Fofão carrying the torch—a cultural nuance that text-based searches would likely overlook.

Limitations and Future Work

This report covers a single task within a larger ongoing project. Future work includes expanding classification categories to specific sporting disciplines and developing new visualization techniques, such as a "videosphere." The authors intend to further explore human-AI collaboration to make large image collections more expressive and navigable for researchers.

Summary updated

BibTeX
@inproceedings{giannella2017observator,
  author = {Julia Giannella and Luiz Velho},
  title = {{ObservatóR!O2016}},
  booktitle = {DH},
  year = {2017},
}
Figure