Auditory Display of Hyperspectral Colon Tissue Images Using Vocal Synthesis Models

Ryan J. Cassidy, Jonathan Berger, Kyogu LeeICAD 2004 · OpenReview

Introduction

Hyperspectral imaging captures vast amounts of diagnostic data beyond human visual perception. While rich in information, visualizing 128-dimensional spectral data per pixel is a significant challenge. Traditional sonification—mapping data to raw parameters like pitch or volume—often results in unintelligible noise. This research, presented at the 2004 International Conference on Auditory Display (ICAD), proposes using human vocal synthesis as a "lens" for medical data. By leveraging the human auditory system's innate ability to categorize speech sounds, the researchers developed a system that converts tissue pathology into intuitive human vowel sounds.

Synthesis Methodology

The core of the system is the transformation of high-dimensional data into parameters for vocal synthesis. The researchers explored two primary synthesis architectures:

  1. Formant-Filter-Based Synthesis (Klatt Model): This approach utilizes a spectrally rich excitation source passed through a series of resonators. These resonators define "formants," the spectral peaks that characterize specific vowels.
  2. Cascaded-Tube-Section Model (Cook Model): This physical modeling approach simulates the vocal tract as a series of acoustic tubes. Wave propagation is modeled using digital waveguides, with Kelly-Lochbaum scattering junctions representing impedance changes between sections of varying radii.
High-level block diagram of the vocal synthesis system, showing the source-filter model.

To bridge the gap between 128-dimensional hyperspectral pixels and synthesis parameters, a dimensionality reduction pipeline was employed. Principal Components Analysis (PCA) reduced the data to 16 dimensions, followed by Local Discriminant Bases (LDB) to reach 5 dimensions. Finally, a nearest-neighbor classifier produced three probability values (abnormal nucleus, normal nucleus, or non-nucleic tissue), which were mapped directly to the radii of the simulated vocal tract or formant frequencies.

The transition from a physical vocal tract shape to a discretized tube model and its corresponding digital waveguide implementation.

Dataset and Implementation

The study utilized hyperspectral datacubes of colon tissue provided by Yale University. The dataset included 15 normal and 46 malignant tissue cubes, each 491 x 652 pixels with 128 spectral bands (440–700 nm). The samples were prepared as microdots on slides, representing tissue from over 300 distinct patients.

The researchers implemented the physical model by creating a new VoicTract C++ class for the open-source Synthesis ToolKit (STK), facilitating broader use of Cook’s model in the sonification community.

Discussion: The Vowel Advantage

The primary innovation lies in "categorical perception." Humans do not merely perceive frequency shifts; they categorize sounds into distinct phonemes. Mapping malignant tissue to an /i/ sound (as in "team") and benign tissue to an /a/ sound (as in "father") allows clinicians to identify outliers through a well-understood perceptual framework.

Furthermore, the system maintains "distance preservation," ensuring that mathematically similar data points sound perceptually similar. To further distinguish classes, the researchers introduced "multi-speaker mapping," where different tissue types are assigned different vocal characteristics (e.g., male vs. child voices), increasing the perceptual distance between diagnostic categories.

Results and Future Directions

Qualitative analysis confirmed that the system produces coherent, speech-like results. Spectrograms of the synthesized output demonstrate distinct harmonic structures rather than chaotic noise. Malignant tissue consistently produced clear /i/ sounds, while benign tissue produced /a/ sounds.

Future work will focus on optimizing the mapping between data and vocal tract parameters through empirical perceptual studies. Additionally, the researchers aim to validate the system's diagnostic efficacy in clinical settings and explore the sonification of even higher-dimensional data by leveraging the human ability to recognize multiple simultaneous vowels.

Summary updated

BibTeX
@inproceedings{cassidy2004auditory,
  author = {Ryan J. Cassidy and Jonathan Berger and Kyogu Lee},
  title = {{Auditory Display of Hyperspectral Colon Tissue Images Using Vocal Synthesis Models}},
  booktitle = {ICAD},
  year = {2004},
  url = {https://openreview.net/forum?id=8qicAwdkGf},
}
Figure