Applications of Neural Networks in Video Signal Processing

John C. Pearson, Clay Spence, Ronald SverdloveNeurIPS 1990

Published by researchers at the David Sarnoff Research Center, this paper explores a practical application of neural networks in what was then considered a "mature" technology: color television. While neural networks (NNs) are often associated with futuristic fields like robotics, the authors argue that embedding them in high-volume consumer electronics is the key to driving down hardware costs and catalyzing the broader field of AI hardware.

The Challenge of Impulse Noise

In the era of analog and early digital TV, "impulse noise"—caused by electric sparks from everyday items like vacuum cleaners or power tools—created distracting horizontal, multi-colored streaks on the screen.

Removing this noise is surprisingly difficult for two primary reasons:

  1. False Positives: Many legitimate parts of a video signal, such as sharp edges or small textural details, look exactly like noise to a simple filter.
  2. Color Complexity: The way color is encoded in a composite video signal makes it hard to simply "average out" the noise without destroying the color fidelity or blurring the underlying image.

The authors sought to create a system that could accurately identify these tiny corrupted segments and replace them with a "best guess" (interpolation) without degrading the rest of the picture.

A Modular Neural Network Approach

To solve this, the researchers moved away from a "one-size-fits-all" network. Instead, they designed a Modular Neural Network System consisting of three distinct Multi-Layer Perceptrons (MLPs), each specialized for a specific part of the pipeline.

Modular Neural Network Architecture
  • The e-net (Energy Filter): This first stage of detection looks at 9 consecutive samples. It is trained to act as a filter that extracts the "energy" of the impulse noise while ignoring the underlying video signal.
  • The d-net (Detector): This network takes the output of the e-net. It looks at a 3x5 grid of samples to decide if a specific pixel is actually noise (outputting a 1) or just a sharp image feature (outputting a 0).
  • The i-net (Interpolator): If the d-net flags a pixel as noisy, the i-net steps in. It looks at the lines above and below the current pixel to calculate a replacement value that blends seamlessly with the surrounding image.

To prove this could work in the real world, they simulated the system on the Princeton Engine, a massively parallel supercomputer capable of 29.3 billion instructions per second (GIPS), using 16-bit fixed-point arithmetic.

Training, Validation, and Key Insights

The networks were trained using a single frame of video corrupted with synthetic AM impulses of five different amplitudes. To ensure the network hadn't just memorized the training data (overfitting), it was evaluated on a completely different video frame.

During development, the authors noted that Mean Squared Error (MSE)—the standard mathematical metric for "closeness"—did not correlate well with how humans actually perceive video quality. Furthermore, they found that a single network struggled to distinguish noise from edges. By using the e-net to pre-process the signal into "noise energy" before the d-net made a classification, they achieved much higher accuracy.

Instead of relying solely on error rates, they used Receiver Operating Characteristic (ROC) curves to validate their models. This allowed them to visualize the critical trade-off between "true positives" (catching the noise) and "false alarms" (blurring the actual image).

Key Results

The modular neural network system demonstrated significant improvements over existing methods:

  • Superior Detection: The neural network detector significantly outperformed the industry-standard median filter. ROC analysis showed the NN could catch more noise with fewer false alarms across all noise amplitudes.
ROC Curves comparing NN and Median Detectors
  • Interpolation Gains: The i-net (interpolator) provided a 1.5 dB improvement in signal quality over conventional interpolation methods.
  • Real-Time Feasibility: The system was successfully demonstrated in real-time on the Princeton Engine, proving that the computational complexity was manageable for high-speed video processing.

Limitations and Future Work

Despite its success, the system had a few limitations. The d-net occasionally missed the very edges (fringes) of noise impulses, requiring the authors to add a manual post-processing step to "expand" the detection zone around flagged pixels.

Additionally, while the NN is technically superior, the authors acknowledged the hurdle of cost-reduction. A median filter is very cheap to build in silicon; the challenge is making a neural network chip that is just as inexpensive. Moving forward, the team planned to investigate "casting" these small networks into specialized, low-cost chips and applying similar NN techniques to other video problems, such as de-interlacing.

Summary updated

BibTeX
@inproceedings{pearson1990applications,
  author = {John C. Pearson and Clay Spence and Ronald Sverdlove},
  title = {{Applications of Neural Networks in Video Signal Processing}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {1990},
}
Figure