Export
Hee Suk Yoon
· 3 previous affiliations
- PhD student · Since 2022
- MS student · 2021–2022
- Intern · 2025
- Georgia Institute of Technology2019–2021
- Undergrad student · 2019–2021
Research Overview
Hee Suk Yoon focuses on the spatial intelligence of vision-language models, model alignment through preference optimization, and robust test-time adaptation. His early work addressed calibration and exposure bias in generative models, gradually shifting toward multimodal reasoning and reinforcement learning from human feedback. Currently, he investigates how vision-language models process spatial relationships and visual transformations to solve complex reasoning problems.
Research Trajectory
His early research addressed fundamental challenges in multimodal learning and model calibration. In 2022, he contributed to work on video-grounded dialogue systems, focusing on the tendency of models to hallucinate text by copying conversation history rather than processing visual inputs. This led to the development of an information-theoretic framework to identify and penalize these shortcut features in Information-Theoretic Text Hallucination Reduction for Video-grounded Dialogue. The following year, he investigated the overconfidence of modern neural networks, introducing a tuning-free trainable calibration measure in ESD: Expected Squared Difference as a Tuning-Free Trainable Calibration Measure to reduce the computational costs associated with hyperparameter tuning on large datasets. During this period, he also explored generative error correction and adaptation in speech processing. To address exposure bias in sentence-level grapheme-to-phoneme conversion, he developed a loss-based sampling method in Mitigating the Exposure Bias in Sentence-Level Grapheme-to-Phoneme (G2P) Transduction, forcing models to practice correcting their own likely mistakes and contributing to the broader area of Scheduled Sampling for Robust Autoregressive Generation. Concurrently, he proposed a parameter-efficient approach to enhance non-native speech recognition in INTapt: Information-Theoretic Adversarial Prompt Tuning for Enhanced Non-Native Speech Recognition, which nudges non-native speech features to resemble native ones without degrading underlying model performance.
In 2024, his focus expanded to test-time adaptation and the integration of visual history in dialogue. He introduced a method to calibrate vision-language models during test-time adaptation by maximizing text feature dispersion in the latent space, detailed in C-TPT: Calibrated Test-Time Prompt Tuning for Vision-Language Models via Text Feature Dispersion. This approach ensures that confidence scores accurately reflect model accuracy without requiring labeled data. He applied similar adaptive principles to speech recognition in unfamiliar environments with LI-TTA: Language Informed Test-Time Adaptation for Automatic Speech Recognition, which leverages external language models to provide contextual corrections and prevent semantically nonsensical phonetic errors. Returning to multimodal dialogue, he collaborated on BI-MDRG: Bridging Image History in Multimodal Dialogue Response Generation. This framework utilizes a novel citation-tagging system that enables AI agents to maintain consistent object identities and ground their responses in specific visual details from earlier in a conversation, bypassing the limitations of text-only summaries.
A significant portion of his recent trajectory centers on refining how large language models are aligned with human preferences. Moving beyond coarse, sequence-level judgments, he introduced a framework for more granular guidance in TLCR: Token-Level Continuous Reward for Fine-grained Reinforcement Learning from Human Feedback. This work advanced the research directions of Token-Level Credit Assignment for Language Model Alignment and Online Reward Modeling for Robust Alignment. Building on this token-level perspective, he developed a zero-overhead alignment method in ConfPO: Exploiting Policy Model Confidence for Critical Token Selection in Preference Optimization. By focusing training exclusively on the high-information tokens the model is least confident about, this approach prevents reward hacking and contributes to the area of Token Level Importance for Selective Optimization. He further extended these principles to complex reasoning tasks in PACR: Progressively Ascending Confidence Reward for LLM Reasoning. By tracking a model's internal confidence at each step to provide a dense reward signal, PACR eliminates the need for external reward models and accelerates training on complex math benchmarks, aligning with the broader field of Automated Process Supervision for Large Language Model Reasoning.
The most recent phase of his work investigates the spatial intelligence and generative capabilities of vision-language and diffusion models. To bridge the gap between discrete image tokens and continuous pixel generation, he co-developed a plug-and-play diffusion decoder in High-Fidelity Text-to-Image Generation from Pre-Trained Vision-Language Models via Distribution-Conditioned Diffusion Decoding, which extracts and calibrates the rich probability distributions hidden inside a model's token prediction process. Addressing the tendency of small vision-language models to prioritize fluent language over visual evidence, he introduced Visual Gradient Steering in Decomposed On-Policy Distillation for Vision-Language Reasoning: Steering Gradients for Visual Grounding, a method that mathematically separates language style from visual grounding during distillation. Finally, he contributed to streamlining the alignment of diffusion models in Diffusion Negative Preference Optimization Made Simple, which leverages the architecture of Classifier-Free Guidance to achieve superior image alignment and safety using a single network, significantly reducing computational overhead.
Current Directions
The most recent papers in the evidence address the fundamental reasoning dynamics and generative alignment of vision-language and diffusion models. Recent work investigates how to overcome the dominance of linguistic priors in multimodal reasoning, ensuring that models prioritize actual visual grounding over fluent but hallucinated text during distillation. Additionally, current research explores streamlined preference optimization for diffusion models, seeking to improve image alignment and safety while minimizing computational overhead. This includes developing techniques to extract and calibrate probability distributions from pre-trained models to enable high-fidelity, continuous visual generation without requiring extensive retraining.
Profile updated
Loading new papers…
Other researchers whose papers cluster into the same research directions — a proxy for overlapping technical interests and shared intellectual neighborhoods.
See something wrong? Report a mistake