I aim to elicit model behavior through the lens of interpretability, which I refer to as Interpretability + X where X includes multimodality, agents, training, and alignment. Previously, I have worked on multimodal language models including reasoning, fact-checking and retrieval.
I received my B.S. in Computer Science and Engineering from Seoul National University. Also, I worked as a software engineer for 3 years at Hyperconnect (acquired by Match Group ), where I developed an human-in-the-loop AI content moderation system.
The Effectiveness and Durability of Pretraining Data Filtering in Protein Language Models
Aaron Maiwald*, Roderick McNeill*, Jaeyoung Lee*, Matthew A. Clarke*, Lennart Justen, Alejandro Acelas, Ivo Andrews, Giles Harper-Donnelly, Adam Moore, Sophie Rose, Oliver M. Crook†, and Felix Jackson†
Pretraining filtering is one of the most common safeguards for open-weight language models. Here we study its effectiveness, durability and scalability using the ESM-2 family of protein language models. We find that filtering viral proteins from the pretraining data meaningfully degrades masked language modelling, mutation effect prediction and long-range contact prediction on viral proteins, while largely preserving performance on non-viral proteins. These effects are concentrated among proteins from human-infecting viruses, and grow as we scale models from 8M to 3B parameters. As models increase in size, we find that stringent filtering becomes more effective. However, fine-tuning on sequences from the removed viral families restores the lost performance. Although the cost to recover performance increases with model scale, even for the largest models it costs about 80 USD of H100 time. This work establishes pretraining filtering as moderately effective at selectively degrading performance on viral proteins, but not durable against fine-tuning with modest computational resources and access to relevant data.
@article{maiwald2026effectivenessdurabilitypretrainingdata,title={The Effectiveness and Durability of Pretraining Data Filtering in Protein Language Models},author={Maiwald, Aaron and McNeill, Roderick and Lee, Jaeyoung and Clarke, Matthew A. and Justen, Lennart and Acelas, Alejandro and Andrews, Ivo and Harper-Donnelly, Giles and Moore, Adam and Rose, Sophie and Crook, Oliver M. and Jackson, Felix},journal={arXiv},year={2026},}
COLM
v1: Learning to Point Visual Tokens for Multimodal Grounded Reasoning
Jiwan Chung*, Junhyeok Kim*, Siyeol Kim, Jaeyoung Lee, Minsoo Kim, and Youngjae Yu
We present v1, a lightweight extension to Multimodal Large Language Models(MLLMs) that enables selective visual revisitation during inference. While current MLLMs typically consume visual input only once and reason purely over internal memory, v1 introduces a simple point-and-copy mechanism that allows the model to dynamically retrieve relevant image regions throughout the reasoning process. This mechanism augments existing architectures with minimal modifications, enabling contextual access to visual tokens based on the model’s evolving hypotheses. To train this capability, we construct v1g, a dataset of 300K multimodal reasoning traces with interleaved visual grounding annotations. Experiments on three multimodal mathematical reasoning benchmarks—MathVista, MathVision, and MathVerse—demonstrate that v1 consistently improves performance over comparable baselines, particularly on tasks requiring fine-grained visual reference and multi-step reasoning. Our results suggest that dynamic visual access is a promising direction for enhancing grounded multimodal reasoning. Code, models, and data will be released to support future research.
@article{chung2025dontlookoncemultimodal,title={v1: Learning to Point Visual Tokens for Multimodal Grounded Reasoning},author={Chung, Jiwan and Kim, Junhyeok and Kim, Siyeol and Lee, Jaeyoung and Kim, Minsoo and Yu, Youngjae},journal={COLM},year={2026},}
AAAI (Oral)
Do Language Models Associate Sound with Meaning? A Multimodal Study of Sound Symbolism
Sound symbolism is a linguistic concept that refers to non-arbitrary associations between phonetic forms and their meanings. We suggest that this can be a compelling probe into how Multimodal Large Language Models (MLLMs) interpret auditory information in human languages. We investigate MLLMs’ performance on phonetic iconicity across textual (orthographic and IPA) and auditory forms of inputs with up to 25 semantic dimensions (e.g., sharp vs. round), observing models’ layer-wise information processing by measuring phoneme-level attention fraction scores. To this end, we present LEX-ICON, an extensive mimetic word dataset consisting of 8,052 words from four natural languages (English, French, Japanese, and Korean) and 2,930 systematically constructed pseudo-words, annotated with semantic features applied across both text and audio modalities. Our key findings demonstrate (1) MLLMs’ phonetic intuitions that align with existing linguistic research across multiple semantic dimensions and (2) phonosemantic attention patterns that highlight models’ focus on iconic phonemes. These results bridge domains of artificial intelligence and cognitive linguistics, providing the first large-scale, quantitative analyses of phonetic iconicity in terms of MLLMs’ interpretability.
@article{jeong2025languagemodelsassociatesound,title={Do Language Models Associate Sound with Meaning? A Multimodal Study of Sound Symbolism},author={Jeong, Jinhong and Lee, Sunghyun and Lee, Jaeyoung and Han, Seonah and Yu, Youngjae},journal={AAAI (Oral)},year={2025},}
EMNLP
How to Train Your Fact Verifier: Knowledge Transfer with Multimodal Open Models
Jaeyoung Lee, Ximing Lu, Jack Hessel, Faeze Brahman, Youngjae Yu, Yonatan Bisk, Yejin Choi, and Saadia Gabriel
Given the growing influx of misinformation across news and social media, there is a critical need for systems that can provide effective real-time verification of news claims. Large language or multimodal model based verification has been proposed to scale up online policing mechanisms for mitigating spread of false and harmful content. While these can potentially reduce burden on human fact-checkers, such efforts may be hampered by foundation model training data becoming outdated. In this work, we test the limits of improving foundation model performance without continual up- dating through an initial study of knowledge transfer using either existing intra- and inter- domain benchmarks or explanations generated from large language models (LLMs).
We evaluate on 12 public benchmarks for fact-checking and misinformation detection as well as two other tasks relevant to content moderation - toxicity and stance detection. Our results on two recent multi-modal fact-checking benchmarks, Mocheg and Fakeddit, indicate that knowledge transfer strategies can improve Fakeddit performance over the state-of-the-art by up to 1.7% and Mocheg performance by up to 2.9%.
@article{lee2024open-fact-verifier,title={How to Train Your Fact Verifier: Knowledge Transfer with Multimodal Open Models},author={Lee, Jaeyoung and Lu, Ximing and Hessel, Jack and Brahman, Faeze and Yu, Youngjae and Bisk, Yonatan and Choi, Yejin and Gabriel, Saadia},journal={EMNLP Findings},year={2024},}