A Synchronized Audio-Visual Multi-View Capture System

Xiangwei Shi, Gara Dorta, Ruud de Jong, Ojas Shirekar, Chirag RamanarXiv 2026 · arXiv

TL;DR

This technical report presents a comprehensive blueprint for an audio-visual multi-view capture system that treats synchronized audio and video as co-equal, first-class signals. By unifying networked machine-vision cameras and multi-channel wireless microphones under a shared hardware timing architecture, the system achieves sub-frame alignment crucial for fine-grained human interaction and conversational AI research.

What Problem Does This Paper Address?

Multi-view capture studios are a staple in computer vision and graphics for high-quality 3D human reconstruction and motion tracking. However, existing pipelines are overwhelmingly video-centric; audio capture is frequently treated as an afterthought, leading to poor alignment or unaddressed clock drift.

This is a major bottleneck for human interaction research. Nuanced conversational behaviors—such as turn-taking, backchannels, interruptions, and co-speech gestures—rely on millisecond-level coordination between speech and body motion. Misalignments can completely obscure these interactive dynamics. The authors address this gap by providing a reproducible, "blueprint-level" guide to engineering a capture space where audio-video synchronization matches the strict rigor of multi-camera synchronization.

Methods

The system is built around a modular $3.5\text{m} \times 3.5\text{m} \times 2.0\text{m}$ aluminum extrusion cubic frame lined with green-screen curtains.

Top and interior views of the modular capture frame.

Its architecture splits into three tightly synchronized subsystems:

  • Video Pipeline: 12 IO Industries Volucam machine-vision cameras equipped with Sony IMX253 global-shutter sensors (capturing at $4096 \times 3008$ resolution up to 68 fps). Shutter synchronization is handled at the hardware level via IEEE-1588 Precision Time Protocol (PTP) over a 10GbE LAN switch using a master-slave topology.
  • Audio Pipeline: A scalable 32-channel Sennheiser wireless microphone array managed by an RME Fireface UFX III audio interface and an M-32 AD converter. All audio channels are locked to a unified timeline using a shared word clock reference with jitter suppression.
  • The Synchronization Bridge: Because cameras (networked PTP domain) and audio interfaces (word-clock domain) operate on heterogeneous time bases, the authors introduce a master Linear Timecode (LTC) generator. The continuous LTC signal is split: one path embeds the timecode directly into the master camera's video metadata, while the other path records it as a discrete audio track on the audio interface. Post-processing alignment is achieved simply by matching these two LTC signals.
PTP synchronization topology among cameras.

Datasets

To validate the system in realistic research workflows, the authors collected two distinct datasets:

  1. Multi-person Conversational Interaction Dataset: Roughly 9 hours of synchronized multi-view video and multi-channel audio featuring 3–6 participants per session. Participants engaged in film discussions, debates, and turn-based games. The data was processed using OpenPose and EasyMocap to yield temporally smooth 3D skeletal trajectories alongside clean, isolated speech.
  2. Single-subject Talking Head Generation Dataset: The studio was reconfigured with Sigma 18–35mm zoom lenses focused closely on a single participant's face and upper torso. This pipeline extracts 3D facial landmarks and 3D gaze vectors, creating a high-fidelity dataset for audio-driven animation.

Key Insights and Ideas

  • Hardware Over Software: Tighter cross-view alignment for fast human motion requires shifting synchronization away from software-triggered commands (which suffer from operating system and network scheduling jitter) into the cameras' dedicated timing hardware.
  • The LTC Split Trick: Bridging networked packets and continuous analog audio samples doesn't require complex network-to-audio clock translation. Recording the same analog LTC signal simultaneously as video metadata and an audio channel creates an uncorruptible, explicit temporal link.
  • Strict Operational Protocols Matter: Perfect temporal sync can still be ruined by visual artifacts. The authors emphasize manual, fixed camera controls—such as a uniform one-shot white balance using a matte white sphere and a strict exposure time limit (3399 µs)—to eliminate view-dependent color shifts and motion blur.

Key Results

  • Inter-Camera Alignment: Under the PTP master-slave configuration, the maximum inter-camera timestamp offset reported during active capture was on the order of a few nanoseconds (up to 6 ns in the best-case setup).
  • End-to-End Audio-Visual Sync: Evaluated using an event-based audiovisual test stimulus (a dropping ball striking a bar), the audio onset consistently occurred less than one video frame earlier than the visual contact. At a 60 fps recording configuration, this translates to a reliable audio lead of < 16.7 ms, confirming sub-frame accuracy.

Limitations and Future Work

  • Budgetary Prioritizations: The authors note that hardware-synchronized cameras and high-end audio interfaces are non-negotiable primary constraints. If working under a strict budget, researchers should prioritize these hardware sync interfaces over maximizing camera counts.
  • Future Scaling: Given an increased budget, the authors recommend increasing the camera density. Adding more viewpoints reduces persistent occlusions during multi-person interactions and inherently improves the robustness of downstream multi-view 3D reconstructions against occasional tracking failures.

Summary updated

Abstract

Multi-view capture systems have been an important tool in research for recording human motion under controlling conditions. Most existing systems are specified around video streams and provide little or no support for audio acquisition and rigorous audio-video alignment, despite both being essential for studying conversational interaction where timing at the level of turn-taking, overlap, and prosody matters. In this technical report, we describe an audio-visual multi-view capture system that addresses this gap by treating synchronized audio and synchronized video as first-class signals. The system combines a multi-camera pipeline with multi-channel microphone recording under a unified timing architecture and provides a practical workflow for calibration, acquisition, and quality control that supports repeatable recordings at scale. We quantify synchronization performance in deployment and show that the resulting recordings are temporally consistent enough to support fine-grained analysis and data-driven modeling of conversation behavior.
BibTeX
@misc{shi2026synchronized,
  author = {Xiangwei Shi and Gara Dorta and Ruud de Jong and Ojas Shirekar and Chirag Raman},
  title = {{A Synchronized Audio-Visual Multi-View Capture System}},
  year = {2026},
  eprint = {2603.23089},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2603.23089},
}
Figure