Neural Interferometry: Image Reconstruction from Astronomical Interferometers Using Transformer-Conditioned Neural Fields

Benjamin Wu, Chao Liu, Benjamin Eckart, Jan KautzAAAI 2022 · OpenReview

Abstract

Astronomical interferometry enables a collection of telescopes to achieve angular resolutions comparable to that of a single, much larger telescope. This is achieved by combining simultaneous observations from pairs of telescopes such that the signal is mathematically equivalent to sampling the Fourier domain of the object. However, reconstructing images from such sparse sampling is a challenging and ill-posed problem, with current methods requiring precise tuning of parameters and manual, iterative cleaning by experts. We present a novel deep learning approach in which the representation in the Fourier domain of an astronomical source is learned implicitly using a neural field representation. Data-driven priors can be added through a transformer encoder. Results on synthetically observed galaxies show that transformer-conditioned neural fields can successfully reconstruct astronomical observations even when the number of visibilities is very sparse.

Introduction

To capture high-resolution images of distant celestial objects, such as the event horizon of a black hole, astronomers utilize Very-Long-Baseline Interferometry (VLBI). By linking multiple telescopes across the globe—as seen with the Event Horizon Telescope (EHT)—researchers can effectively create an Earth-sized virtual lens.

Unlike traditional cameras, these telescope arrays do not capture pixels directly. Instead, they record sparse, continuous points in the spatial frequency (Fourier) domain, termed "visibilities." Reconstructing a 2D image from these sparse samples is a highly ill-posed inverse problem. Traditional approaches, such as the CLEAN algorithm, are often labor-intensive, requiring expert manual tuning and iterative processing. Furthermore, standard deep learning models like Convolutional Neural Networks (CNNs) struggle with this data because telescope measurements do not align with a discrete pixel grid.

Methods

Neural Interferometry addresses these challenges by operating entirely in the continuous Fourier domain. The architecture combines two key innovations:

  1. Neural Fields (Coordinate-based MLPs): Rather than predicting a grid of pixels, the model uses a Multi-Layer Perceptron (MLP) to represent the continuous frequency spectrum. For any given frequency coordinate $(u, v)$, the MLP predicts the corresponding complex visibility.
  2. Transformer Encoder: To incorporate data-driven priors without retraining for every new observation, a Transformer is used. It processes the sparse visibility measurements and their coordinates as a set of input tokens, leveraging the architecture's inherent permutation invariance.
  3. FiLM Conditioning: The Transformer outputs latent tokens that modulate the MLP layers via Feature-wise Linear Modulation (FiLM). This conditions the neural field on specific observations, enabling single-shot, feed-forward reconstruction.
The Neural Interferometry architecture: a Transformer encoder conditions a coordinate-based MLP to reconstruct the continuous Fourier spectrum.

Key Results

The proposed method was evaluated using synthetic observations of galaxies from the Galaxy10 datasets, simulated to match the 8-telescope EHT configuration. The Transformer-conditioned neural field significantly outperformed both traditional algorithms and deep learning baselines.

  • Quantitative Performance: The model achieved a Structural Similarity Index (SSIM) of 0.968, vastly exceeding the CLEAN algorithm (0.174) and a U-Net baseline (0.786).
  • Qualitative Fidelity: As shown in the comparison below, the method faithfully recovers the shape and orientation of galaxies, whereas traditional methods often leave heavy artifacts or struggle with extended emissions.
  • Efficiency: By utilizing a feed-forward pass at inference time, the model is significantly faster than the iterative processes required by traditional pipelines.
Qualitative comparison of the reconstruction against ground truth (GT) and the standard 'dirty image' resulting from simple Fourier inversion.

Conclusion

Neural Interferometry demonstrates that combining Transformers with coordinate-based neural fields provides a powerful framework for astronomical image reconstruction. While the current work assumes well-calibrated visibilities, it sets the stage for future research into handling raw, uncalibrated atmospheric data. This approach offers a faster, more accurate, and data-driven alternative to the manual tuning currently required in the field of radio astronomy.

Summary updated

Abstract

Astronomical interferometry enables a collection of telescopes to achieve angular resolutions comparable to that of a single, much larger telescope. This is achieved by combining simultaneous observations from pairs of telescopes such that the signal is mathematically equivalent to sampling the Fourier domain of the object. However, reconstructing images from such sparse sampling is a challenging and ill-posed problem, with current methods requiring precise tuning of parameters and manual, iterative cleaning by experts. We present a novel deep learning approach in which the representation in the Fourier domain of an astronomical source is learned implicitly using a neural field representation. Data-driven priors can be added through a transformer encoder. Results on synthetically observed galaxies show that transformer-conditioned neural fields can successfully reconstruct astronomical observations even when the number of visibilities is very sparse.
BibTeX
@inproceedings{wu2022neural,
  author = {Benjamin Wu and Chao Liu and Benjamin Eckart and Jan Kautz},
  title = {{Neural Interferometry: Image Reconstruction from Astronomical Interferometers Using Transformer-Conditioned Neural Fields}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2022},
  url = {https://openreview.net/forum?id=nhV0He4SrCN},
}
Figure