The Software Defined Media Ontology for Music Events

Ray Atarashi, Takuro Sone, Yu Komohara, Manabu Tsukada, Takashi Kasuya, Hiraku Okumura, Masahiro Ikeda, Hiroshi EsakiSAAM@ISWC 2018 · DOI

TL;DR

This paper introduces the Software Defined Media (SDM) Ontology, a framework for managing and sharing object-based audio and video data from music events. By structuring this data as Linked Open Data (LOD), the authors enable flexible content creation and interactive playback applications, paving the way for richer, more personalized media experiences.

What Problem Does This Paper Address?

The increasing ubiquity of recording devices, especially in smartphones, has led to an explosion of audio-visual data. Traditionally, this data is processed as raw audio channels or simple bitmaps, limiting its potential for sophisticated analysis and interactive use. While object-based media and 3D representations are gaining traction, there's a need for a standardized way to describe and manage the complex metadata associated with these recordings, especially in dynamic environments like music events. This includes details about the performers, instruments, recording equipment, venue, and the recording process itself. Without such a framework, sharing and reusing this data for novel applications becomes a significant challenge.

Methods

The core of this work is the Software Defined Media (SDM) Ontology. This ontology is designed using RDF (Resource Description Framework) to represent music event data as Linked Open Data (LOD). The ontology is structured around five key elements:

  • SDMEvent: Represents the overall event.
  • Target: Describes entities being recorded (e.g., musicians, instruments, audience).
  • Context: Captures the environment and situation (e.g., program details, venue).
  • Recorder: Details the recording devices used.
  • MediaObject: Represents the actual recorded audio, video, or other data.

The authors leverage existing vocabularies like schema.org where appropriate and define new classes and properties to capture the specifics of music event recordings. They then implement a system that converts recorded data into LOD using this ontology and store it in a SPARQL endpoint for external access. Two applications, SDM360 (a tablet-based interactive viewer) and LiVRation (a VR application), are developed to demonstrate the utility of this LOD-based approach.

Datasets

The paper describes two primary datasets collected for this research:

  • Concert at Keio University's Collegium Musicum Early Music Academy: This dataset comprises recordings of acoustic performances with up to 24 musicians, including various instruments. Multiple microphones and cameras were used to capture individual instruments and the overall soundscape.
  • Recording in Billboard Live Tokyo: This dataset includes recordings from jazz band concerts and a J-pop artist performance. Various microphone arrays, 360-degree cameras, and audience microphones were employed.

These datasets were used to populate the SDM ontology and validate the developed applications. The paper does not introduce new benchmark datasets but rather uses these real-world recordings to demonstrate the ontology's capabilities.

Key Insights and Ideas

  • Unified Data Representation: The SDM Ontology provides a consistent and structured way to represent diverse audio-visual data from music events, bridging the gap between raw recordings and usable content.
  • Linked Open Data for Media: By treating media data as LOD, the authors enable powerful cross-referencing and integration with external knowledge bases like Wikidata, enriching the metadata and facilitating discovery.
  • Enabling Interactive Experiences: The ontology's detailed representation of object-based audio and video allows for applications like SDM360 and LiVRation, where users can freely navigate perspectives, isolate instruments, and experience immersive audio-visual environments.
  • Flexibility and Extensibility: The ontology is designed to be extensible, allowing for the addition of new recording device types or media formats as technology evolves.
  • Bridging Recording and Application Development: The framework facilitates collaboration by allowing different parties to contribute recordings and develop applications based on the shared data, fostering innovation.

Key Results

The paper demonstrates the successful implementation of the SDM Ontology and its application in two interactive systems:

  • SDM360: This tablet application allows users to interactively play back 3D content from music events, change perspectives, and enable/disable individual instrument audio objects.
  • LiVRation: This VR application provides an immersive experience using HMDs, allowing users to explore virtual spaces, stream high-resolution audio, and access contextual information.

The authors show that the LOD generated using the SDM ontology is sufficient for these diverse applications, proving the ontology's effectiveness in managing and utilizing complex media data. They also demonstrate how the ontology can be used to query external data sources like Wikidata for richer context.

Limitations and Future Work

  • Manual Description of Positions: The current system relies on manual description of microphone and camera positions. Future work aims to automate this process using 3D positioning techniques.
  • Integration with Existing Ontologies: While schema.org is used, further integration with specialized music and event ontologies could enhance the system's semantic richness.
  • Broader Use Cases: The paper suggests exploring more "minus one" content scenarios for musical practice and integrating data from multiple, distributed recording points.
  • Scalability and Performance: While not explicitly detailed, the performance and scalability of the SPARQL endpoint and the applications with very large datasets would be an area for further investigation.

Summary updated

BibTeX
@inproceedings{atarashi2018software,
  author = {Ray Atarashi and Takuro Sone and Yu Komohara and Manabu Tsukada and Takashi Kasuya and Hiraku Okumura and Masahiro Ikeda and Hiroshi Esaki},
  title = {{The Software Defined Media Ontology for Music Events}},
  booktitle = {SAAM@ISWC},
  year = {2018},
  doi = {10.1145/3243907.3243915},
}
Figure