Plasma: a scripting language for processing media streams

Tao Zhu, Pavel Korshunov, Bing Liu, Wei Tsang OoiProceedings of SPIE, the International Society for Optical Engineering/Proceedings of SPIE 2007 · DOI

TL;DR

This paper introduces Plasma, a new scripting language designed for the rapid development of applications that process live and stored media streams. Plasma treats media streams as first-class objects, offering unique features for handling network conditions and spatial composition, making complex multimedia tasks more accessible.

What Problem Does This Paper Address?

The increasing prevalence of streaming media in areas like education, entertainment, and surveillance has created a demand for applications that go beyond simple playback. These applications often require complex manipulations such as mixing, filtering, and composing multiple media streams, especially for live feeds. Existing tools and libraries, however, tend to be either too specialized for stored media or lack the flexibility to handle the unique challenges of live streams, such as network variability and real-time processing. This gap makes it difficult and time-consuming to develop sophisticated multimedia applications.

Methods

Plasma is presented as a scripting language, built upon the Tcl/Tk framework. This choice leverages Tk's strengths in GUI development and event handling, allowing for intuitive spatial arrangement and event-driven callbacks. The core of Plasma's approach lies in its abstractions:

  • Media Objects: A unified abstraction for live streams, stored files, and analog devices. These objects can be nested, enabling hierarchical composition (e.g., a video stream containing another video stream as a picture-in-picture).
  • Event Objects: These represent phenomena that occur at specific times, triggered by user input, timers, network conditions, or media content analysis (like motion detection).
  • Indiva Middleware: Plasma integrates with Indiva to interact with analog media devices (cameras, VCRs), treating them as regular media objects.

Plasma's syntax is designed to be familiar to Tk users, facilitating spatial arrangement with commands like mpack and event binding with mbind. A key differentiator is its exposure of underlying network conditions (e.g., loss rate) to the programmer, enabling adaptive application logic.

Datasets

Not discussed in the paper.

Key Insights and Ideas

  • Media Streams as First-Class Citizens: Treating media streams as fundamental objects, rather than just data, is crucial for building expressive multimedia applications. This allows for direct manipulation and composition.
  • Bridging Live and Stored Media: Plasma's ability to seamlessly mix live and stored media streams simplifies many common scenarios, such as inserting pre-recorded content into live broadcasts.
  • Leveraging Tk's Strengths: The decision to base Plasma on Tcl/Tk provides a familiar and powerful environment for GUI development and event handling, significantly lowering the barrier to entry for developers.
  • Network-Aware Scripting: Exposing network statistics directly to the script allows for dynamic adaptation of media streams based on real-time network conditions, a capability often missing in simpler tools.
  • Spatial Composition is Key for Live Media: The paper emphasizes that while temporal manipulations are common for stored media, spatial arrangements (like picture-in-picture or side-by-side layouts) are more critical for live streaming applications.

Key Results

The paper demonstrates Plasma's effectiveness through several examples and two applications:

  • Examples: Scripts are shown for capturing camera feeds to disk, arranging multiple streams side-by-side, creating picture-in-picture effects, triggering actions based on motion detection, and adapting frame rates based on network loss. These examples highlight the conciseness and power of Plasma for complex tasks.
  • VJOnline Application: A tool for video jockeys and web-cast directors, implemented with approximately 300 lines of code (240 for media processing). It supports transitions, picture-in-picture, and text effects, showcasing Plasma's rapid prototyping capabilities.
  • SLIME Application: A Stored/Live Media Editor designed for intuitive video composition. Despite its complexity, the core application is about 8000 lines of Tcl/Tk code, with Plasma handling all video processing. Notably, when Plasma gained support for live streams, SLIME automatically gained streaming video capabilities with minimal code changes, demonstrating Plasma's generality.

Limitations and Future Work

  • Performance: While Plasma is designed for ease of use, its performance for highly demanding, real-time applications might require further optimization. The paper mentions that Plasma is still under development and work is ongoing to optimize performance.
  • Generic Streams: The authors express interest in implementing "generic streams" in Plasma. This would extend its capabilities beyond audio/video to handle other data streams (like text or numbers), enabling richer applications such as displaying scrolling stock tickers on video.
  • Distributed Applications: Currently, Plasma applications are designed to run on a single host, communicating with other servers. Future work might explore making Plasma applications themselves distributed.

Summary updated

Abstract

Media streaming has found applications in many domains such as education, entertainment, communication and video surveillance. Many of these applications require non-trivial manipulations of media streams, beyond the usual capture/playback operations supported by typical multimedia software and tools. To support rapid development of such applications, we have designed and implemented a scripting language called Plasma. Plasma treats media streams as first-class objects, and caters to the characteristic differences between stored media files and live media streams. In this paper, we illustrate the design and features of Plasma through several small examples, and describe two example applications that we developed on top of Plasma. These two applications demonstrate that using Plasma, complex applications that compose, mix, and filter multimedia streams can be written with relatively little effort.
BibTeX
@inproceedings{zhu2007plasma,
  author = {Tao Zhu and Pavel Korshunov and Bing Liu and Wei Tsang Ooi},
  title = {{Plasma: a scripting language for processing media streams}},
  booktitle = {Proceedings of SPIE, the International Society for Optical Engineering/Proceedings of SPIE},
  year = {2007},
  doi = {10.1117/12.707248},
}
Figure