Image and Information

Frank NielsenarXiv 2016 · OpenReview

Abstract

A well-known old adage says that "A picture is worth a thousand words!" (attributed to the Chinese philosopher Confucius ca 500 years BC). But more precisely, what do we mean by information in images? And how can it be retrieved effectively by machines? We briefly highlight these puzzling questions in this column. But first of all, let us start by defining more precisely what is meant by an "Image."

Introduction

While the notion that a picture is worth a thousand words is intuitive, the technical definition of "information" within an image is surprisingly complex. Is an image merely a grid of RGB values, or is it the semantic meaning we extract from it? This work bridges the gap between the physical acquisition of light and the high-level interpretation of visual data. It explores how digital images are constructed (and often reconstructed) and how the field of computer vision has shifted from manually engineering features to letting neural networks discover them implicitly.

The Digital Imaging Pipeline

Modern digital sensing follows a fundamental pipeline that governs how we perceive visual data. It begins with acquisition, where sensors (photodiodes) capture and discretize light. Most consumer cameras employ a single sensor overlaid with a Bayer filter—a color filter array. Because each pixel only records one color (Red, Green, or Blue), the hardware must perform demosaicing (interpolation) to reconstruct a full RGB image.

The Bayer filter array (a) and its decomposition into the captured image (b) and individual red (c), green (d), and blue (e) channels.

Beyond standard photography, this pipeline extends to various modalities such as Ultrasound, MRI, and Hyperspectral imaging, where raw sensor data is transformed into an "abstract" image for human or machine consumption.

Information Extraction: From Hand-Crafted Features to Deep Learning

The field of computer vision has undergone a paradigm shift in how it handles information. Traditional methods relied on explicit, hand-crafted features—such as edges or specific color segments—to identify objects. In contrast, Deep Learning (DL) utilizes hierarchical architectures to learn implicit features across stacked layers, mimicking the processing of the human visual cortex.

Key Insights: Images as Reconstructions

A critical realization is that we rarely interact with "raw" data. Every digital image is a reconstruction derived from a discrete signal sampled at regular positions. The theoretical limit of visual information is described by the 7D plenoptic function $f(X, Y, Z, \theta, \phi, \lambda, t)$, which accounts for every light ray's position, orientation, wavelength, and time.

A comparison between the original image (f) and the reconstructed image (g) following demosaicing, highlighting potential artifacts and resolution loss.

Furthermore, the "information" required from an image is task-dependent. A soccer-playing robot may only need to segment a red ball, whereas an autonomous vehicle requires a massive fusion of data from multiple sensors to navigate safely. Human perception also plays a role; our vision system processes high frequencies (edges) and low frequencies (textures) differently, a principle exploited in "hybrid images" that change appearance based on viewing distance.

Applications and Future Challenges

These principles enable powerful applications:

  • Photogrammetry: Reconstructing high-precision 3D models from 2D photo collections.
  • Semantic Segmentation: Partitioning images into meaningful categories (e.g., road, pedestrian) for robotics.
  • Automated Annotation: Using DL for real-time tasks like "smile shutters" without manual pixel-level definitions.

However, a significant hurdle remains: certifiability. While deep learning is revolutionary, its performance in novel, high-stakes environments (like autonomous driving) is not yet 100% certifiable. Future progress depends on better information fusion—integrating diverse sensor data to create robust systems capable of handling visual ambiguity.

Summary updated

Abstract

A well-known old adage says that {\em "A picture is worth a thousand words!"} (attributed to the Chinese philosopher Confucius ca 500 years BC). But more precisely, what do we mean by information in images? And how can it be retrieved effectively by machines? We briefly highlight these puzzling questions in this column. But first of all, let us start by defining more precisely what is meant by an "Image."
BibTeX
@misc{nielsen2016image,
  author = {Frank Nielsen},
  title = {{Image and Information}},
  year = {2016},
  eprint = {1602.01228},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1602.01228},
}
Figure