Export
Yong Jae Lee
· 4 previous affiliations
- University of Wisconsin–MadisonSince 2021
- Full Professor · Since 2021
- Adobe Systems (United States)Since 2025
- Principal Researcher · Since 2025
- University of California, Davis2014–2021
- Associate Professor · 2014–2021
- Cruise2020–2021
- Researcher · 2020–2021
Research Overview
Yong Jae Lee focuses on computer vision, multimodal representation learning, and generative modeling. His trajectory spans from unsupervised visual disentanglement and grounded image generation to the efficient scaling of large multimodal models. Currently, his work investigates robust evaluation frameworks for vision-language models, cross-modal knowledge transfer, and generalized forensics for synthetic media.
Research Trajectory
Early in his observed trajectory, Lee's work established foundations in several related areas of visual representation learning. He contributed extensively to the research direction of Overcoming Spurious Contextual Correlations in Visual Models and the broader area of Contextual Co-occurrence Bias in Visual Representation Learning. His efforts during this period also advanced methodologies in areas such as Suppressing Salient Features for Robust Representation Learning and Discriminative Part Bias in Weakly Supervised Localization.
Building on these themes, his 2019 research addressed the area of Decoupling Foreground Objects from Environmental Context with the introduction of FineGAN: Unsupervised Hierarchical Disentanglement for Fine-Grained Object Generation and Discovery. This framework utilized information theory to associate latent codes with background, object shape, and appearance, enabling the unsupervised hierarchical generation of fine-grained object categories without relying on manual annotations.
As generative AI rapidly advanced, Lee's focus expanded to spatial control and grounding in diffusion models. In Gligen: Open-set grounded text-to-image generation, he and his collaborators developed a method to condition pre-trained text-to-image models on grounding inputs like bounding boxes. By injecting this information through gated mechanisms while freezing the original weights, the model preserved its vast conceptual knowledge while gaining spatial controllability. This spatial control was further refined in Generate Anything Anywhere in Any Scene, which introduced PACGen to overcome spatial entanglement, allowing users to precisely place personalized objects within generated scenes. Alongside generation, he investigated the utility of these models for downstream tasks in Diversify, Don't Fine-Tune: Scaling Up Visual Recognition Training with Synthetic Images. This work explored the scalability challenges of training ImageNet classifiers with large volumes of synthetic data, highlighting performance degradation when synthetic images outnumber real ones.
Simultaneously, Lee addressed the efficiency and personalization of Large Multimodal Models (LMMs). To enable LMMs to recognize specific subjects without forgetting general knowledge, his group proposed Yo'LLaVA: Your Personalized Language and Vision Assistant, which embedded user-specific concepts into a small set of learnable tokens. To tackle the computational inefficiency of processing dense visual scenarios, he guided the development of a nested approach to visual representation. Initially presented in Matryoshka Multimodal Models (2024) and formally published as Matryoshka Multimodal Models at ICLR 2025, this architecture allows LMMs to dynamically adjust the number of visual tokens at inference time, trading off computational cost for detail within a single set of weights.
In 2025, a significant portion of his research pivoted toward image forensics, specifically the detection of AI-generated media. Recognizing that traditional detectors fail to generalize across different generative architectures, he co-authored Towards Universal Fake Image Detectors that Generalize Across Generative Models, which leveraged the feature space of pre-trained vision-language models to achieve robust detection without explicit fake-detection training. He further refined forensic methodologies in Aligned Datasets Improve Detection of Latent Diffusion-Generated Images, demonstrating that training detectors on aligned pairs of real images and their autoencoder reconstructions forces the model to focus purely on generative artifacts rather than semantic shortcuts.
Most recently, in 2026, Lee's work has focused on the reasoning capabilities and evaluation integrity of vision-language models. He explored cross-modal knowledge distillation in Large Language Model Teaches Visual Students: Cross-Modality Transfer of Fine-Grained Conceptual Knowledge, introducing LaViD to transfer conceptual signatures from text-only LLMs to vision models via generated multiple-choice questions. To rigorously test multimodal reasoning, his team introduced DocHop: Benchmarking Out-of-domain Multi-hop Reasoning in Information-Dense Documents, a benchmark requiring models to aggregate data across interleaved charts and text. Finally, addressing the critical issue of benchmark validity, he contributed to Contamination Detection for VLMs Using Multi‑Modal Semantic Perturbations, a method that uses generative AI to create perturbed test questions, effectively exposing models that rely on memorized test-set leakage rather than genuine reasoning.
Current Directions
The most recent papers in the evidence indicate a strong focus on the rigorous evaluation and enhancement of reasoning in Large Multimodal Models (LMMs). Recent work investigates how to bridge the conceptual gap between text-only language models and vision models, utilizing cross-modal distillation techniques to transfer fine-grained semantic knowledge without relying on paired image-text datasets. Additionally, Lee is actively addressing the integrity of multimodal benchmarks. This includes developing complex, multi-hop reasoning tasks for information-dense documents and creating robust contamination detection methods to identify test-set leakage, ensuring that models are evaluated on genuine reasoning capabilities rather than memorized data.
Profile updated
Loading new papers…
Other researchers whose papers cluster into the same research directions — a proxy for overlapping technical interests and shared intellectual neighborhoods.
See something wrong? Report a mistake