Sofia Malanchuk

Moscow Institute of Physics and Technology · Skolkovo Institute of Science and Technology
2 previous affiliations
3Machine learning papers tracked by Lacuna
Is this you?

Research Overview

Sofia Malanchuk focuses on the development of information-theoretic methods for high-dimensional data analysis and the interpretability of deep neural networks. Her work primarily addresses the computational and theoretical challenges of estimating entropy and mutual information in complex systems, utilizing techniques such as lossy compression, autoencoders, and normalizing flows. By bridging representation learning and information theory, her research seeks to provide stable, scalable tools for monitoring the internal dynamics of neural architectures and quantifying the information content of large-scale datasets.

Research Trajectory

The available research record indicates a trajectory focused on the refinement of information-theoretic estimators, beginning with a focus on entropy estimation and expanding into the analysis of deep learning dynamics. In 2023, the work centered on the fundamental problem of calculating entropy in high-dimensional spaces, where traditional non-parametric estimators often fail due to the curse of dimensionality. This period established a methodological foundation in using lossy compression as a proxy for information measurement.

In 2024, the research scope evolved to apply these estimation techniques to the Information Bottleneck (IB) framework. This shift moved the focus from static dataset analysis to the dynamic process of neural network training, specifically tracking how information is filtered and compressed across successive layers of convolutional networks. Most recently, the work has transitioned toward generative modeling approaches, specifically using Normalizing Flows to transform complex data distributions into structured latent spaces. This evolution represents a move from purely compression-based estimation toward coordinate-transformation methods that allow for more stable and analytically tractable mutual information calculations.

Methodological Focus

A recurring theme in Malanchuk’s work is the mitigation of variance and instability in high-dimensional information estimation. The methodologies employed can be categorized into two primary approaches:

  1. Compression-Based Estimation: This approach utilizes autoencoders to project high-dimensional inputs into lower-dimensional latent representations. By applying lossy compression, the research demonstrates that it is possible to derive theoretically consistent entropy estimates that remain robust even as the input dimensionality increases. This method has been applied both to synthetic benchmarks and to standard image datasets to validate its empirical utility.
  2. Latent Space Warping via Normalizing Flows: To overcome the limitations of traditional neural mutual information estimators, which often suffer from high computational overhead and instability, recent work has focused on Normalizing Flows. By warping complex data into a "tridiagonal" Gaussian distribution, the research enables the calculation of mutual information using closed-form expressions in the latent space, significantly reducing the variance associated with sample-based estimation.

Representative Work

The following papers represent the core of the research trajectory observed between 2023 and 2024.

In High-Dimensional Dataset Entropy Estimation via Lossy Compression, Malanchuk contributed to the development of a novel entropy estimation framework. The paper proposes using an autoencoder-based compression step to facilitate entropy calculation in high-dimensional contexts. The authors provide theoretical proofs for the relevance of this estimation method and demonstrate its performance through experiments on synthetic data and the MNIST digits dataset, showing that lossy compression can serve as an effective tool for capturing the underlying information structure of complex data.

In collaborative research published in 2024, Information Bottleneck Analysis of Deep Neural Networks via Lossy Compression, Malanchuk participated in the application of the Information Bottleneck principle to realistic convolutional neural networks. The study addresses the long-standing difficulty of estimating mutual information between high-dimensional layer outputs and input/output variables. By incorporating a lossy compression step before estimation, the authors were able to observe the "fitting" and "compression" phases of the learning process in architectures where traditional estimators typically fail. This work provides empirical evidence for the IB theory's applicability to modern deep learning models.

Malanchuk also participated in the development of a new estimation method detailed in Mutual Information Estimation via Normalizing Flows. This paper introduces MIENF, a method designed to provide accurate and stable mutual information estimates for high-dimensional data. The approach uses Normalizing Flows to transform complex data into a latent space characterized by a specific tridiagonal Gaussian distribution. This transformation allows for the use of closed-form expressions for mutual information, resulting in lower computational overhead and greater theoretical consistency compared to standard neural estimators. The method addresses the variance and instability issues that frequently plague information-theoretic analysis in machine learning.

The current body of work reflects a concentrated effort to provide the mathematical and algorithmic tools necessary for a more rigorous information-theoretic understanding of high-dimensional data and the models that process it. While the evidence is currently focused on a two-year window, it demonstrates a clear progression toward increasing methodological sophistication in the service of model interpretability and data analysis.

Profile updated

See something wrong? Report a mistake