Dealing with Synchronization and Timing Variability in the Playback of Interactive Session Recordings

Nelson R. Manohar, Atul PrakashACM Multimedia 1995 · DOI

TL;DR

This paper tackles the challenge of faithfully replaying interactive session recordings, especially when the replay environment introduces timing variations. They introduce an adaptive scheduling algorithm and a toolkit called REPLAYKIT to manage these timing differences, ensuring synchronized playback of audio and window streams. This is crucial for asynchronous collaboration where users might miss live sessions and need to catch up later.

What Problem Does This Paper Address?

In collaborative environments, users often work together in real-time on applications (synchronous collaboration). However, this can be restrictive. The authors' previous work introduced a paradigm for asynchronous collaboration where users can record and replay interactive sessions. These recordings, called "session objects," capture user interactions, audio annotations, and application state. The core problem this paper addresses is ensuring that these session objects can be replayed accurately and synchronously, even when the computer used for replay has different performance characteristics (timing variability) than the recording machine. This variability can arise from differences in CPU speed, load conditions, and timer inaccuracies, leading to desynchronized audio and visual streams, which degrades the replay experience and its usefulness for collaboration.

Methods

The paper proposes an adaptive scheduling algorithm to handle timing variability during the replay of session objects. The core idea is to adjust the playback rate of individual streams (specifically, the window stream, treated as a "slave" to the audio "master" stream) on the fly.

Here's a breakdown of their approach:

  • Media Model: They model audio and window streams as discrete, time-based streams. The audio stream is treated as continuous and synchronous, while the window stream is stateful, aperiodic, discrete, and asynchronous.
  • Synchronization Mechanism: They use a "master/slave" model where the audio stream is the master and the window stream is the slave. Synchronization is achieved through "synchronization events" posted at the end of significant events (like audio frames).
  • Adaptive Scheduling: This is the key innovation. Instead of a fixed schedule, their algorithm dynamically adjusts the inter-event delays within the window stream's scheduling interval.
  • Time Deformations: They introduce a "compensation factor" (ω) that scales the inter-event delays. If the window stream is falling behind the audio stream, the delays are compressed (ω < 1), effectively speeding up replay. If the window stream is ahead, the delays are expanded (ω > 1), slowing it down.
  • Statistical Process Control: The algorithm uses historical asynchrony data and statistical guidelines to detect trends and forecast future deviations. This allows for proactive adjustments rather than reactive ones.
  • Per-Stream Scheduling: Each stream has its own scheduler, allowing for independent adjustments based on its specific timing requirements and tolerances.
  • REPLAYKIT Toolkit: They developed an object-oriented toolkit that allows applications to become "replay-aware." This toolkit provides the REPLAYABLE object class, enabling applications to inherit session capture and replay functionality and access their synchronization infrastructure.

Datasets

Not discussed in the paper. The authors describe experiments using a "MAC DRAW-like drawing application" and a "TEXT editor application" retrofitted with their toolkit. The session objects were created by recording approximately 250 seconds of voice-annotated user interactions with these applications.

Key Insights and Ideas

  • Asynchronous Replay is Tricky: The paper highlights that simply replaying events isn't enough. The inherent timing variability of computer systems makes direct, uncompensated replay lead to desynchronization, especially for complex, stateful streams like window interactions.
  • Slave-Initiated Synchronization: Shifting the synchronization initiation from the master (audio) to the slave (window) stream reduces overhead on the master and allows for more flexible adjustments.
  • Statistical Control is Key: Relying on statistical trends and forecasts, rather than just immediate asynchrony measurements, allows for smoother and more effective compensation of timing variability. This prevents abrupt jumps in replay speed.
  • Application-Layer Control is Necessary: For faithful replay, especially in collaborative contexts, synchronization needs to be managed at the application level, not just at the OS or hardware level, to account for application-specific behaviors and timing tolerances.
  • REPLAYKIT's Utility: The toolkit demonstrates a practical way to integrate sophisticated replay and synchronization capabilities into existing applications without requiring deep OS-level modifications.

Key Results

  • Protocol P3 Performance: Their adaptive scheduling protocol (P3) significantly outperformed simpler protocols (P1: laissez-faire, P2: blocking) in terms of reducing both the mean and variance of inter-stream asynchrony across various background load conditions (lower, same, higher).
  • Reduced Asynchrony: For protocol P3, the mean asynchrony was about half the mean scheduling interval duration, and the standard deviation was about one-third.
  • Robustness to Load: The adaptive mechanism maintained stable asynchrony levels across different load conditions, indicating its robustness.
  • Qualitative Improvement: Observers judged P3 to be qualitatively superior, with better audio continuity and less drift in window events compared to P1 and P2, especially under varying load conditions.
  • Differential Scheduling Benefits: Using a differential scheduler (periodical lookups) was found to be statistically better than an absolute scheduler (frequent lookups) in reducing variability.

Limitations and Future Work

  • Audio Frame Size: While 16KB audio frame sizes worked well, smaller sizes led to appreciable thread scheduling overheads, affecting audio quality. Larger sizes didn't offer significant performance gains.
  • 2-WAY Protocol Issues: Protocol P4 (2-way blocking) was found to be unacceptable due to gaps introduced in audio playback.
  • Future Work:
    • Extend the framework to support video streams.
  • Explore the paradigm in broader collaborative environments.
  • Apply the REPLAYKIT toolkit to the UARC and MEDRAD projects.

Summary updated

BibTeX
@inproceedings{manohar1995dealing,
  author = {Nelson R. Manohar and Atul Prakash},
  title = {{Dealing with Synchronization and Timing Variability in the Playback of Interactive Session Recordings}},
  booktitle = {ACM Multimedia},
  year = {1995},
  doi = {10.1145/217279.215088},
}
Figure