Zhen Zhu

Research Overview

Zhen Zhu focuses on computer vision and generative modeling, specifically addressing spatial reasoning, semantic segmentation, and the optimization of large-scale multimodal models. Zhu's research has evolved from developing foundational datasets and localization techniques for aerial imagery to designing efficient attention mechanisms for scene parsing and exploring consistency in multimodal image generation. Recently, the work has shifted toward the analysis of learning dynamics in large multimodal models (LMMs), with a focus on mitigating catastrophic forgetting and improving sequential fine-tuning.

Research Trajectory

The early phase of Zhu's research (2017–2018) was centered on the challenges of object detection in specialized domains, such as remote sensing and scene text. This period involved addressing Resolving Spatial Ambiguity in Dense Object Detection and Precise Localization for Arbitrarily Oriented Objects. A significant contribution during this time was participation in the development of DOTA: A Large-scale Dataset for Object Detection in Aerial Images, which provided a benchmark for identifying objects with extreme scale and orientation variance. This work necessitated the use of Multi-Scale Architectures for Extreme Scale Variance in Remote Sensing and techniques for Overlapping Sliding Windows for High Resolution Image Analysis.

Between 2018 and 2020, Zhu's focus transitioned toward semantic segmentation and the synthesis of non-stationary textures. As a second author, Zhu contributed to Rotation-Sensitive Regression for Oriented Scene Text Detection, which proposed a dual-branch network to resolve the incompatibility between classification and regression in oriented text detection. This period also saw the development of Asymmetric Non-local Neural Networks for Semantic Segmentation, where Zhu served as the first author. This work introduced the Asymmetric Pyramid Non-local Block (APNB) to address the high computational and memory costs associated with standard non-local modules. Additionally, Zhu participated in research on Mitigating Boundary Artifacts and Spatial Bias in Convolutional Networks and Maintaining Word Coherence in Sparse Scene Text Detection.

From 2022 to 2024, Zhu's research moved into generative adversarial networks (GANs) and human-centric image generation. As first author, Zhu led the development of Progressive and Aligned Pose Attention Transfer for Person Image Generation, which utilized a sequence of transfer blocks to maintain appearance and shape consistency during pose deformation. This was followed by work on Consistent Multimodal Generation via A Unified GAN Framework, which focused on the simultaneous generation of consistent RGB, depth, and surface normal maps. During this phase, Zhu also participated in the development of lightweight frameworks for video face swapping and single-image concept insertion for text-to-image models.

In the most recent period (2025–2026), Zhu has focused on the pedagogical aspects of Large Multimodal Models (LMMs). The first-author paper How to Teach Large Multimodal Models New Skills investigates the dynamics of sequential fine-tuning and identifies that apparent forgetting in LMMs is often linked to shifts in output token distributions rather than a total loss of prior knowledge.

Current Directions

Zhu's current research is primarily concerned with the stability and efficiency of large-scale foundation models. A central theme is the development of robust tuning recipes that allow LMMs to acquire new capabilities without degrading their performance on existing benchmarks. This involves analyzing the drift in internal representations and identifying specific model components, such as self-attention projections, that are most conducive to stable learning.

Furthermore, Zhu continues to explore generative priors in the context of image restoration and enhancement. This includes participation in zero-shot methods for low-light image enhancement that leverage diffusion priors while minimizing semantic hallucinations. The overarching goal of these current efforts is to bridge the gap between high-fidelity generative capacity and the practical requirements of consistency and reliability in multimodal systems.

Representative Work

Asymmetric Non-local Neural Networks for Semantic Segmentation

In Asymmetric Non-local Neural Networks for Semantic Segmentation, Zhu addresses the prohibitive computational and memory costs of the non-local module, which is widely used to capture long-range dependencies in semantic segmentation. As first author, Zhu introduces the Asymmetric Pyramid Non-local Block (APNB) and the Asymmetric Fusion Non-local Block (AFNB). The APNB utilizes a pyramid sampling module to reduce the number of keys and values in the attention mechanism, effectively lowering the complexity from $O(N^2)$ to $O(N \times M)$, where $M$ is significantly smaller than $N$. The AFNB further adapts this approach to fuse features across different levels of the network. The results demonstrate that these modules improve performance on benchmarks such as Cityscapes and ADE20K while maintaining a manageable computational footprint.

Progressive and Aligned Pose Attention Transfer for Person Image Generation

As first author of Progressive and Aligned Pose Attention Transfer for Person Image Generation, Zhu proposes a generative adversarial network designed for high-fidelity pose transfer. The core of the model is a progressive generator composed of pose-attentional transfer blocks (PATB) and aligned pose-attentional transfer blocks (APATB). These blocks iteratively model the relationship between the source and target poses using an attention mechanism, which allows the model to preserve the appearance and shape consistency of the subject. The progressive nature of the generator enables the handling of complex deformations that are often problematic for single-step transfer models. The work is validated through experiments on the DeepFashion and Market-1501 datasets, showing improvements in photorealism and consistency.

Consistent Multimodal Generation via A Unified GAN Framework

In Consistent Multimodal Generation via A Unified GAN Framework, Zhu investigates the challenge of generating multiple consistent image modalities, such as RGB, depth, and surface normals, from a single generative model. Serving as first author, Zhu builds upon the StyleGAN3 architecture by implementing a shared backbone with modality-specific branches. The framework introduces per-modality fidelity discriminators and a cross-modality consistency discriminator to ensure that the generated outputs are not only realistic but also physically aligned with one another. The research also provides a training recipe for extending the model to new domains, demonstrating its utility in generating consistent multimodal data for complex scenes.

How to Teach Large Multimodal Models New Skills

In the 2026 paper How to Teach Large Multimodal Models New Skills, Zhu explores the problem of catastrophic forgetting during the sequential fine-tuning of LMMs. As first author, Zhu conducts a systematic study of five target skills across three model families, monitoring performance on eight held-out benchmarks. The research identifies that forgetting is often a manifestation of a measurable shift in the output token distribution, which can be identified using a counting-bias probe. Based on these findings, Zhu proposes two robust tuning recipes: updating only the self-attention projection layers and employing a specific learning rate schedule. These methods allow the models to retain prior abilities while successfully acquiring new skills.

Rotation-Sensitive Regression for Oriented Scene Text Detection

As the second author of Rotation-Sensitive Regression for Oriented Scene Text Detection, Zhu contributes to a solution for the incompatibility between text presence detection and oriented bounding box regression. The paper argues that sharing features between these two tasks leads to degraded performance because classification requires rotation-invariant features, while regression requires rotation-sensitive features. The proposed method utilizes two separate network branches with distinct designs to extract features tailored to each task. This approach significantly improves the accuracy of oriented scene text detection on standard benchmarks like ICDAR2015 and COCO-Text.

Non-stationary Texture Synthesis by Adversarial Expansion

In Non-stationary texture synthesis by adversarial expansion, Zhu, acting as second author, addresses the synthesis of textures that exhibit large-scale structures or spatial variance. Traditional example-based synthesis methods often fail on such non-stationary textures. The proposed approach uses a GAN trained to double the spatial extent of texture blocks from a specific exemplar. Once the fully convolutional generator is trained, it can expand the size of an entire texture while maintaining its structural characteristics. This method provides a flexible way to generate large-scale, inhomogeneous textures that are visually consistent with the original sample.