Wonjae (Dan) Kim
김원재 · Lead Research Scientist · Multimodal AI
At TwelveLabs, I lead the Marengo & Search team, where we build video-native multimodal representations and search systems. Most recently, we launched Marengo 3.5, a multimodal embedding model designed around the temporal structure of video.
I’m a co-first author of ViLT (cited: 3184), an early work on efficient vision-language modeling. Previously, I was a research scientist at Naver AI LAB and Kakao, and I hold an M.Sc. and B.Sc. from Seoul National University.
My current research focuses on:
- Video-native Multimodal Representation Learning (video, audio, image, text, documents)
- Large-scale Embedding & Search Systems
- User Behavior Modeling for Search
I also write about information, AI tools, and coordination. Recent essays include Speak Concisely, Write Verbosely and The Theory of Consensus Reset Cost Optimization.
At TwelveLabs, we’re hiring researchers and engineers to work on multimodal embeddings and video search. Open role in Seoul → · Coffee chat
news
We launched Marengo 3.5, a video-native multimodal embedding model built around the temporal structure of video.
TwelveLabs launches Pegasus 1.5, a state-of-the-art video-language model for understanding and reasoning over long-form video.
TwelveLabs releases Marengo 3.0, a new standard for foundation models that understand the world in all its complexity.
One CVPR-2025 EVAL-FoMo 2 Workshop paper: Emergence of Text Readability in Vision Language Models.
I’ve started a new chapter at TwelveLabs!
latest posts
[번역] 나는 내 재능을 어제에 묻을 수밖에 없다
AI가 자신보다 커널을 더 잘 작성하게 될 날을 앞두고, 한 DeepSeek 개발자가 일과 애정, 오픈소스의 미래를 돌아본 글의 한국어 번역.
The Theory of Consensus Reset Cost Optimization
How the World Breaks Monopolies, and Why Different Assets Break Differently
Speak Concisely, Write Verbosely
LLM 시대, 간결함의 황금률이 무너지는 순간
selected publications
- NeurIPSLearning Dynamics of Attention: Human Prior for Interpretable Machine Reasoning32nd Conference on Neural Information Processing Systems (NeurIPS 2019)Without relevant human priors, neural networks may learn uninterpretable features. We propose Dynamics of Attention for Focus Transition (DAFT) as a human prior for machine reasoning. DAFT is a novel method that regularizes attention-based reasoning by modelling it as a continuous dynamical system using neural ordinary differential equations. As a proof of concept, we augment a state-of-the-art visual reasoning model with DAFT. Our experiments reveal that applying DAFT yields similar performance to the original model while using fewer reasoning steps, showing that it implicitly learns to skip unnecessary steps. We also propose a new metric, Total Length of Transition (TLT), which represents the effective reasoning step size by quantifying how much a given model's focus drifts while reasoning about a question. We show that adding DAFT results in lower TLT, demonstrating that our method indeed obeys the human prior towards shorter reasoning paths in addition to producing more interpretable attention maps. Our code is available at https://github.com/kakao/DAFT.
@inproceedings{kim2019learning, title = {Learning Dynamics of Attention: Human Prior for Interpretable Machine Reasoning}, author = {Kim, Wonjae and Lee, Yoonho}, booktitle = {32nd Conference on Neural Information Processing Systems (NeurIPS 2019)}, pages = {6019--6030}, year = {2019}, arxiv = {1905.11666}, } - ICMLLong talkViLT: Vision-and-Language Transformer Without Convolution or Region Supervision38th International Conference on Machine Learning (ICML 2021)Vision-and-Language Pre-training (VLP) has improved performance on various joint vision-and-language downstream tasks. Current approaches to VLP heavily rely on image feature extraction processes, most of which involve region supervision (e.g., object detection) and the convolutional architecture (e.g., ResNet). Although disregarded in the literature, we find it problematic in terms of both (1) efficiency/speed, that simply extracting input features requires much more computation than the multimodal interaction steps; and (2) expressive power, as it is upper bounded to the expressive power of the visual embedder and its predefined visual vocabulary. In this paper, we present a minimal VLP model, Vision-and-Language Transformer (ViLT), monolithic in the sense that the processing of visual inputs is drastically simplified to just the same convolution-free manner that we process textual inputs. We show that ViLT is up to tens of times faster than previous VLP models, yet with competitive or better downstream task performance. Our code and pre-trained weights are available at https://github.com/dandelin/vilt.
@inproceedings{pmlr-v139-kim21k, title = {ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision}, author = {Kim*, Wonjae and Son*, Bokyung and Kim, Ildoo}, booktitle = {38th International Conference on Machine Learning (ICML 2021)}, pages = {5583--5594}, year = {2021}, editor = {Meila, Marina and Zhang, Tong}, volume = {139}, series = {Proceedings of Machine Learning Research}, month = {18--24 Jul}, publisher = {PMLR}, arxiv = {2102.03334}, video = {https://icml.cc/virtual/2021/oral/9492}, } - ECCVOralHYPE: Hyperbolic Entailment Filtering for Underspecified Images and Texts18th European Conference on Computer Vision (ECCV 2024)In an era where the volume of data drives the effectiveness of self-supervised learning, the specificity and clarity of data semantics play a crucial role in model training. Addressing this, we introduce HYPerbolic Entailment filtering (HYPE), a novel methodology designed to meticulously extract modality-wise meaningful and well-aligned data from extensive, noisy image-text pair datasets. Our approach leverages hyperbolic embeddings and the concept of entailment cones to evaluate and filter out samples with meaningless or underspecified semantics, focusing on enhancing the specificity of each data sample. HYPE not only demonstrates a significant improvement in filtering efficiency but also sets a new state-of-the-art in the DataComp benchmark when combined with existing filtering techniques. This breakthrough showcases the potential of HYPE to refine the data selection process, thereby contributing to the development of more accurate and efficient self-supervised learning models. Additionally, the image specificity ϵi can be independently applied to induce an image-only dataset from an image-text or image-only data pool for training image-only self-supervised models and showed superior performance when compared to the dataset induced by CLIP score.
@misc{kim2024hype, title = {HYPE: Hyperbolic Entailment Filtering for Underspecified Images and Texts}, author = {Wonjae Kim and Sanghyuk Chun and Taekyung Kim and Dongyoon Han and Sangdoo Yun}, year = {2024}, arxiv = {2404.17507}, booktitle = {18th European Conference on Computer Vision (ECCV 2024)}, archiveprefix = {arXiv}, primaryclass = {cs.CV}, }