Learning Through Text-Image Pairs and Image Sequences

Qicheng LaoSpectrum 2019

Abstract

Many machine learning systems are biologically inspired, drawing from the architecture of the human brain (ANNs) or the visual cortex (CNNs). Beyond structural inspiration, this research explores a higher-level cognitive parallel: multi-view learning. Humans naturally integrate information from diverse sources—such as visual cues, textual descriptions, and temporal sequences—to make optimal decisions. This paper addresses challenges in text-to-image synthesis, cell phenotype classification, histopathological diagnosis, and disease progression by modeling the underlying connections between distinct feature sets. By treating image sequences and text-image pairs as multi-view data, we demonstrate that explicitly learning these relationships leads to superior performance in both generative and diagnostic tasks.

Introduction

In the natural world, intelligence rarely operates on isolated data points. A human identifying a bird considers its physical appearance, its song, and written descriptions of its habitat. Traditional machine learning, however, often treats these inputs as independent or fails to distinguish between different types of information within a single modality.

This research identifies a fundamental gap in how AI handles "multi-view" data. Whether it is the distinction between the content of a text description and the artistic style of an image, or the relationship between low-magnification and high-magnification medical scans, the connections between these views are often as important as the data itself. This paper proposes several architectures designed to bridge these gaps across diverse domains.

Methodology

1. Text-to-Image Synthesis: Dual Adversarial Inference (DAI)

A primary challenge in text-to-image synthesis is the conflation of "content" (the semantic description) and "style" (visual attributes like lighting, position, or background not specified in text). To address this, we introduce Dual Adversarial Inference (DAI). Unlike standard Generative Adversarial Networks (GANs) that rely on a single noise vector, DAI utilizes an inference mechanism (an encoder) to separate the latent space into two distinct variables: c (content derived from text) and z (style derived from the image). A cycle-consistency loss is implemented to ensure that the model can reconstruct the original style and content, forcing the generator to respect the boundaries between what is described and what is visually inferred.

2. Segmentation-Free Multi-Channel Learning

In high-throughput drug screening, traditional pipelines require a computationally expensive "segmentation" step to outline individual cells before classification. We propose a multi-view approach that treats raw, five-channel fluorescent microscopy images as distinct views. By using ResNet-based architectures directly on these channels, the model learns to classify cell phenotypes without the need for manual or automated segmentation, preserving the spatial context of the entire image.

3. Case-Based Histopathology and Disease Progression

Medical diagnosis often relies on sequences. In histopathology, pathologists examine tissue at multiple magnifications (e.g., 40x to 400x). We redefine a "case" as a sequence of these views, using a fusion model to integrate features across scales simultaneously. Similarly, for Disease Progression Learning (DPL), we treat disease stages as a continuous sequence rather than independent categories. By combining CNNs for feature extraction with Long Short-Term Memory (LSTM) networks, and employing a cyclic training strategy, the model learns the temporal logic of disease evolution.

Experimental Results

Generative Performance

The DAI framework was evaluated on the Oxford-102, CUB-200, and COCO datasets. It achieved a significant improvement in Fréchet Inception Distance (FID) scores, which measure both image quality and diversity. On the CUB-200 dataset, the model reached an FID of 18.41, outperforming the previous state-of-the-art HDGAN (20.60). This confirms that separating style from content allows for more precise and diverse image generation.

Medical and Biological Accuracy

  • Cell Phenotyping: The segmentation-free multi-channel approach achieved 98.2% accuracy on three-class compound classification using the BBBC022v1 dataset, demonstrating that segmentation is not a prerequisite for high-accuracy biological screening.
  • Cancer Diagnosis: On the BreaKHis dataset, the case-based magnification fusion reached 91.48% case-level accuracy, significantly higher than models restricted to single-magnification views.
  • Diabetic Retinopathy: The DPL approach provided a 3.3% accuracy boost in staging disease progression compared to standard classification methods, proving the value of modeling the sequential nature of clinical stages.

Discussion and Insights

A key finding of this research is that "disentanglement" can be achieved without explicit labels. In the DAI model, the system learns to separate style from content purely by comparing the text view to the image view. This addresses a common flaw in GANs where the "noise" vector is often ignored by the generator; by requiring an inference step, the model is forced to utilize the style vector to achieve reconstruction.

Furthermore, the success in medical domains highlights the importance of contextual diagnostics. By treating different magnifications or disease stages as a unified sequence, the AI mimics the holistic approach of human experts, leading to more robust and clinically relevant outcomes.

Conclusion and Future Work

This thesis demonstrates that multi-view learning provides a powerful framework for both synthesis and analysis. While the current work shows strong results at lower resolutions (64x64) and shorter sequences, future research should focus on scaling these disentanglement techniques to high-resolution images and longer, more complex clinical histories. The Dual Adversarial Inference framework also holds potential for expansion into other multi-modal pairs, such as audio-visual or speech-to-text integration.

Summary updated

Abstract

Many machine learning systems for artificial intelligence are biologically inspired, for example, the artificial neural networks (ANNs) have similar architecture as human brains, and convolutional neural networks (CNNs) are inspired by the observations from early study on animal's visual cortex system. The above two examples (ANNs and CNNs) are inspirations at the level of creating fundamental tools (e.g., neural networks) for a machine learning system. Another level of inspirations can come from the way human learn or respond that builds on top of the existing powerful learning tools, i.e., brains. In this thesis, we will focus on another type of inspiration that also belongs to the second level. It is based on the common practice that for an efficient learning or an optimal decision, human integrate all sources of available information in multiple views and leverage the reasoning of the underlying connections among them, i.e., multi-view learning. We address several problems in both medical and non-medical domains, including text-to-image synthesis, cell phenotype classification, histopathological malignancy diagnosis and disease progression learning, from the perspective of multi-view learning with an emphasis on learning the underlying connections among the multiple distinct feature sets representing the given multi-view data (i.e., image sequences in a unimodal setting and text-image pairs in a multi-modal setting).
BibTeX
@inproceedings{lao2019learning,
  author = {Qicheng Lao},
  title = {{Learning Through Text-Image Pairs and Image Sequences}},
  booktitle = {Spectrum},
  year = {2019},
}
Figure