The Active Sensing Testbed

Lee Stearns, Neil Fendley, Ashley J. LlorensAAAI 2021 · DOI

Abstract

The Active Sensing Testbed (AST) is a modular software framework designed to move machine perception beyond static image recognition and toward "active" world-view reasoning. By providing a flexible architecture to chain together multi-modal sensors, real-time analytics, and robotic controls, AST allows researchers to build systems that do not just see the world, but actively investigate it to fill in knowledge gaps. This paper outlines the AST architecture and demonstrates its utility through various computer vision tasks.

1. Introduction

Most state-of-the-art computer vision (CV) algorithms are "narrowly trained," excelling at specific tasks in controlled environments but often failing in unpredictable real-world scenarios. These models are typically passive, processing data without the ability to request more information. True machine perception requires active sensing: the ability for a system to take action—such as zooming a camera or changing a sensor's position—to test a hypothesis or resolve ambiguity. The AST addresses the engineering hurdles of integrating hardware, low-level processing, and high-level reasoning by providing a standardized, modular framework.

2. System Architecture and Methods

The AST is built on a server-client architecture designed for flexibility and ease of use. The central server acts as a hub, ingesting data from multiple sensors, such as ceiling-mounted pan-tilt-zoom (PTZ) cameras.

The AST system architecture, showing the flow of data from sources through a router to modular workflows and a RESTful API for client interaction.

2.1 Workflows and Modularity

Users define "graphs" that connect data sources to sequences of operations. For example, a workflow might take a raw video feed, apply a transformation, and then run a detection algorithm. The system provides both a REST API and a Python library, allowing researchers to subscribe to data feeds or issue commands programmatically. This modularity allows researchers to quickly swap out different algorithms or sensors without rewriting the entire system.

2.2 Modular Analytics

The framework is designed to integrate "off-the-shelf" algorithms. The authors demonstrated this by plugging in established models for semantic segmentation (Mask R-CNN), style transfer (CycleGAN), and human pose estimation.

Demonstration of modular analytics, including semantic segmentation and pose estimation, applied to a public video feed from Times Square.

3. Key Insights and Results

One significant innovation in AST is the use of generative AI as a pre-processing step. By transforming a night-time feed into a "day-time" version using CycleGAN before running detection, the system can utilize models trained only on daylight data in a wider variety of conditions.

3.1 Qualitative Successes

The paper serves as a proof of concept for the AST architecture, showcasing several successes:

  • Real-time Face Swapping: Successfully blending a subject's face with a target in a live stream.
  • Dynamic Object Removal: Using semantic segmentation and adaptive background modeling to identify and "erase" people in real-time, filling in the background seamlessly.
  • World-View Reasoning: The system can maintain an internal model of the world to understand object permanence, such as tracking an object even when it is temporarily hidden behind a barrier.

4. Limitations and Future Work

While the style-transfer results are compelling, they are not yet robust across all environments. Additionally, while the framework supports the data necessary for high-level reasoning (like the "frisbee vs. football" exchange), creating the logic to handle complex object permanence remains an open research challenge. Future work will focus on scaling these reasoning capabilities and deploying the AST software in diverse locations to facilitate broader collaborative research.

Summary updated

BibTeX
@inproceedings{stearns2021active,
  author = {Lee Stearns and Neil Fendley and Ashley J. Llorens},
  title = {{The Active Sensing Testbed}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2021},
  doi = {10.1609/aaai.v35i18.18027},
}
Figure