A User Interface for Knowledge Acquisition From Video

Henry LiebermanAAAI 1994

In the mid-1990s, the development of "expert systems" faced a significant hurdle known as the "knowledge acquisition bottleneck." This gap between a domain expert's practical knowledge and a programmer's technical implementation often led to inefficient or inaccurate systems. In the 1994 paper "A User Interface for Knowledge Acquisition From Video," Henry Lieberman of the MIT Media Lab proposed a novel solution: allowing experts to teach computers directly by annotating and interacting with digitized video.

The Knowledge Acquisition Bottleneck

The primary challenge in AI at the time was that experts are often better at performing tasks than describing them in formal logic. Traditionally, a knowledge engineer would interview an expert and manually translate their actions into code—a process prone to errors and omissions. While video captures expert performance perfectly, it remains "semantically opaque" to computers. Lieberman’s system aimed to bridge this gap by providing a user interface where humans could easily imbue video pixels with semantic meaning.

System Architecture and Workflow

The proposed system extends Mondrian, a teachable graphical editor, to incorporate video processing. The workflow consists of three primary stages:

  1. Frame Selection: The user identifies "salient frames" from a video that represent the state of a device before and after a specific action.
  2. Graphical Annotation: The user defines objects within the video by selecting rectangular regions. These regions are then organized into a part/whole hierarchy using lines and text labels. For instance, a user might label a specific area as a "Plug Connector" belonging to a "Plug Panel" on a "Circuit Board."
  3. Programming by Demonstration (PbD): Instead of writing code, the user demonstrates the desired action in a 2.5D graphical environment. To teach a "Disassemble" command, the user simply moves the image snippet of a component away from its base.
Figure 1: Graphical annotation of a circuit board, defining the part/whole hierarchy of components.

The system utilizes Incremental Explanation-Based Generalization. As the user demonstrates actions, the system generates underlying Lisp code, generalizing specific mouse movements into high-level rules applicable to new, similar objects.

The Power of "Fake" Simulations

A key insight of Lieberman’s work is the concept of a "fake" simulation. Rather than requiring complex 3D CAD models, the system uses rectangular "snips" of video frames as functional icons. These snippets are sufficient for the computer to reason about spatial relationships—such as "above," "below," or "connected to"—enabling effective procedural learning without the overhead of full 3D modeling.

Furthermore, the interface creates an interactive learning loop. Unlike batch machine learning, this system allows for immediate feedback. If the system generalizes a rule incorrectly, the user sees the error on-screen and can provide an instant correction.

Results and Multimodal Outputs

The system was successfully tested on procedures such as circuit board disassembly. Once the system "learned" a procedure from one video, it could apply that logic to new images of different boards, provided the user labeled the corresponding parts.

Figure 2: A visual storyboard generated by the system, illustrating the steps of a disassembly procedure.

One of the most significant achievements was the generation of multimodal outputs from a single demonstration:

  • Executable Code: Lisp expressions representing the learned procedure.
  • Visual Storyboards: A sequence of captioned frames documenting the steps for human reference.
  • Natural Language: Template-based English descriptions of the actions performed.

Discussion and Limitations

While pioneering, the system faced limitations. The manual annotation process was noted as tedious, a task Lieberman suggested could eventually be handled by emerging computer vision algorithms. Additionally, the 2.5D approach struggled with complex 3D issues like perspective changes or total occlusion. Despite these hurdles, the work laid the groundwork for modern interactive machine learning and automated documentation systems.

Summary updated

BibTeX
@inproceedings{lieberman1994user,
  author = {Henry Lieberman},
  title = {{A User Interface for Knowledge Acquisition From Video}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {1994},
}
Figure