profile소개
I build models that let machines understand video.
기계가 영상을 이해하게 하는 모델을 만듭니다.
current work지금 하는 일Director of Research at TwelveLabs, leading research across its video foundation modelsTwelveLabs의 Director of Research로, 회사의 여러 영상 파운데이션 모델 연구를 이끕니다
I lead research and productization across TwelveLabs’ model families. I run hiring across research, built the team, and help set the direction of research and product. I explore highly reusable video representations grounded in structure and understanding. One level up, the same idea runs how the company works with AI: policy, context, agents.
TwelveLabs의 여러 모델 패밀리의 연구와 제품화를 이끕니다. 리서치 채용을 맡아 팀을 꾸렸고, 리서치와 제품의 방향을 함께 세웁니다. 비디오의 구조와 이해를 토대로 재사용성 높은 표현을 탐구하고 있습니다. 같은 생각을 회사 단위로 올려, AI와 일하는 방식을 정책, 맥락, 에이전트로 설계합니다.
marengoMarengoMarengo 3.0 and 3.5: led research and technical directionMarengo 3.0과 3.5: 연구와 기술 방향을 이끌었습니다
Research: Marengo 3.5 splits structure from meaning, with MRL (512 dims beat Gemini Embedding 2’s 3,072) and opt-in ProLIP-based (cited: 29) uncertainty. Product: Marengo 3.0: composed search, sports understanding; 3.5: visual docs. Training: academia’s multi-stage training targets benchmarks; I optimized our stages to shorten development cycles and train on real video and queries.
연구: Marengo 3.5는 구조와 의미를 분리하고, MRL 512차원으로 Gemini Embedding 2의 3,072차원을 앞서며 ProLIP (인용 29) 기반 선택형 불확실성을 냅니다. 제품: Marengo 3.0은 합성 질의와 스포츠 이해, 3.5는 시각 문서를 다룹니다. 학습: 학계는 여러 단계로 나눠 학습하고 벤치마크 점수를 겨냥합니다. 저는 학습 단계를 최적화해 개발 주기를 줄였고, 실제 영상과 질의로 학습했습니다.
before이전 경력Seven years at Kakao and NAVER AI Lab카카오와 NAVER AI Lab에서 7년
Before TwelveLabs, I spent seven years in industry research at Kakao and NAVER AI Lab, where I co-first-authored ViLT (cited: 3,200), an early efficient vision-language transformer now part of Hugging Face Transformers. Its VQA checkpoint was the library’s default visual question answering model and has been downloaded more than 6 million times. I have published 19 peer-reviewed papers and hold an M.Sc. and B.Sc. from Seoul National University.
TwelveLabs에 오기 전에는 카카오와 NAVER AI Lab에서 7년 동안 산업계 연구자로 일했습니다. 그때 공동 제1저자로 쓴 ViLT (인용 3,200)는 효율적인 비전-언어 트랜스포머의 초기 사례 가운데 하나로, 지금은 Hugging Face Transformers에 들어 있습니다. VQA 모델은 이 라이브러리의 시각 질의응답 기본 모델이었고, 내려받기가 600만 회를 넘었습니다. 동료 심사를 거친 논문 19편을 발표했고, 서울대학교에서 학사와 석사 학위를 받았습니다.
At TwelveLabs, we’re hiring researchers and engineers to work on multimodal embeddings and video search. Open role in Seoul →
TwelveLabs에서 멀티모달 임베딩과 영상 검색을 다룰 연구자와 엔지니어를 채용하고 있습니다. 서울 채용 공고 →
selected publications대표 논문
- 2024ECCVOralHYPE: Hyperbolic Entailment Filtering for Underspecified Images and TextsW. Kim, S. Chun, T. Kim, D. Han, S. Yuncited 인용 29In an era where the volume of data drives the effectiveness of self-supervised learning, the specificity and clarity of data semantics play a crucial role in model training. Addressing this, we introduce HYPerbolic Entailment filtering (HYPE), a novel methodology designed to meticulously extract modality-wise meaningful and well-aligned data from extensive, noisy image-text pair datasets. Our approach leverages hyperbolic embeddings and the concept of entailment cones to evaluate and filter out samples with meaningless or underspecified semantics, focusing on enhancing the specificity of each data sample. HYPE not only demonstrates a significant improvement in filtering efficiency but also sets a new state-of-the-art in the DataComp benchmark when combined with existing filtering techniques. This breakthrough showcases the potential of HYPE to refine the data selection process, thereby contributing to the development of more accurate and efficient self-supervised learning models. Additionally, the image specificity ϵi can be independently applied to induce an image-only dataset from an image-text or image-only data pool for training image-only self-supervised models and showed superior performance when compared to the dataset induced by CLIP score.
@misc{kim2024hype, title = {HYPE: Hyperbolic Entailment Filtering for Underspecified Images and Texts}, author = {Wonjae Kim and Sanghyuk Chun and Taekyung Kim and Dongyoon Han and Sangdoo Yun}, year = {2024}, arxiv = {2404.17507}, booktitle = {18th European Conference on Computer Vision (ECCV 2024)}, archiveprefix = {arXiv}, primaryclass = {cs.CV}, }Top 5 by the citing paper's own citation count. “Builds heavily on” is Semantic Scholar's influential-citation label; self-citations are left out. Semantic Scholar, as of Nov 14, 2026.후속 연구 자신의 인용 수가 많은 순으로 5편입니다. 크게 기댔다는 표시는 Semantic Scholar의 ‘영향력 있는 인용’ 기준이고 자기 인용은 뺐습니다. Semantic Scholar 기준, 2026년 11월 14일 현재.
- CLIPLoss and Norm-Based Data Selection Methods for Multimodal Contrastive Learning2024 · Neural Information Processing Systemscited인용 31
- Multilingual Diversity Improves Vision-Language Representations2024 · Neural Information Processing Systemscited인용 19
- Filter Like You Test: Data-Driven Data Filtering for CLIP Pretraining2025 · Neural Information Processing Systemscited인용 7
- LongProLIP: A Probabilistic Vision-Language Model with Long Context Text2025 · arXiv.orgcited인용 5
- ReSpec: Relevance and Specificity Grounded Online Filtering for Learning on Video-Text Data Streams2025 · Computer Vision and Pattern Recognitioncited인용 3
- 2021ICMLLong talkViLT: Vision-and-Language Transformer Without Convolution or Region SupervisionW. Kim*, B. Son*, I. Kimcited 인용 3,200Vision-and-Language Pre-training (VLP) has improved performance on various joint vision-and-language downstream tasks. Current approaches to VLP heavily rely on image feature extraction processes, most of which involve region supervision (e.g., object detection) and the convolutional architecture (e.g., ResNet). Although disregarded in the literature, we find it problematic in terms of both (1) efficiency/speed, that simply extracting input features requires much more computation than the multimodal interaction steps; and (2) expressive power, as it is upper bounded to the expressive power of the visual embedder and its predefined visual vocabulary. In this paper, we present a minimal VLP model, Vision-and-Language Transformer (ViLT), monolithic in the sense that the processing of visual inputs is drastically simplified to just the same convolution-free manner that we process textual inputs. We show that ViLT is up to tens of times faster than previous VLP models, yet with competitive or better downstream task performance. Our code and pre-trained weights are available at https://github.com/dandelin/vilt.
@inproceedings{pmlr-v139-kim21k, title = {ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision}, author = {Kim*, Wonjae and Son*, Bokyung and Kim, Ildoo}, booktitle = {38th International Conference on Machine Learning (ICML 2021)}, pages = {5583--5594}, year = {2021}, editor = {Meila, Marina and Zhang, Tong}, volume = {139}, series = {Proceedings of Machine Learning Research}, month = {18--24 Jul}, publisher = {PMLR}, arxiv = {2102.03334}, video = {https://icml.cc/virtual/2021/oral/9492}, }Top 5 by the citing paper's own citation count. “Builds heavily on” is Semantic Scholar's influential-citation label; self-citations are left out. Semantic Scholar, as of Nov 14, 2026.후속 연구 자신의 인용 수가 많은 순으로 5편입니다. 크게 기댔다는 표시는 Semantic Scholar의 ‘영향력 있는 인용’ 기준이고 자기 인용은 뺐습니다. Semantic Scholar 기준, 2026년 11월 14일 현재.
- LayoutLMv3: Pre-training for Document AI with Unified Text and Image Masking2022 · ACM Multimediacited인용 1,037
- Multimodal Chain-of-Thought Reasoning in Language Models2023 · Trans. Mach. Learn. Res.cited인용 996
- FILIP: Fine-grained Interactive Language-Image Pre-Training2021 · International Conference on Learning Representationscited인용 929
- Visual Programming: Compositional visual reasoning without training2022 · Computer Vision and Pattern Recognitioncited인용 823
- VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Experts2021 · Neural Information Processing Systemscited인용 807
- 2019NeurIPSLearning Dynamics of Attention: Human Prior for Interpretable Machine ReasoningW. Kim, Y. Leecited 인용 12Without relevant human priors, neural networks may learn uninterpretable features. We propose Dynamics of Attention for Focus Transition (DAFT) as a human prior for machine reasoning. DAFT is a novel method that regularizes attention-based reasoning by modelling it as a continuous dynamical system using neural ordinary differential equations. As a proof of concept, we augment a state-of-the-art visual reasoning model with DAFT. Our experiments reveal that applying DAFT yields similar performance to the original model while using fewer reasoning steps, showing that it implicitly learns to skip unnecessary steps. We also propose a new metric, Total Length of Transition (TLT), which represents the effective reasoning step size by quantifying how much a given model's focus drifts while reasoning about a question. We show that adding DAFT results in lower TLT, demonstrating that our method indeed obeys the human prior towards shorter reasoning paths in addition to producing more interpretable attention maps. Our code is available at https://github.com/kakao/DAFT.
@inproceedings{kim2019learning, title = {Learning Dynamics of Attention: Human Prior for Interpretable Machine Reasoning}, author = {Kim, Wonjae and Lee, Yoonho}, booktitle = {32nd Conference on Neural Information Processing Systems (NeurIPS 2019)}, pages = {6019--6030}, year = {2019}, arxiv = {1905.11666}, }
writing글
- 안에서 재는 모양Measuring the Shape from Within
곡면 위의 존재가 바깥 없이 곡률을 아는 방법으로, 샌드박스 속 AI가 자기 세계의 형태를 추론하는 일을 생각해 본다How a being on a curved surface can learn its curvature with no outside to look from, and what that says about an AI in a sandbox inferring the shape of its own world
- The Theory of Consensus Reset Cost Optimization합의 리셋 비용 최적화 이론
How the World Breaks Monopolies, and Why Different Assets Break Differently세상은 독점을 어떻게 부수는가, 그리고 왜 자산마다 부서지는 방식이 다른가
- Speak Concisely, Write Verbosely말할 때는 여전히 간결하게, 쓸 때는 풍부하게
LLM 시대, 간결함의 황금률이 무너지는 순간In the age of LLMs, the moment the golden rule of brevity breaks
log소식
TwelveLabs releases Pegasus 1.6, adding first-person video understanding, stronger entity recognition, native image analysis, and sharper visual comprehension.
TwelveLabs가 Pegasus 1.6을 출시했습니다. 1인칭 영상 이해, 더 정확해진 인물·사물 인식, 이미지 직접 분석, 더 정밀한 시각 이해가 더해졌습니다.
We launched Marengo 3.5, a video-native multimodal embedding model built around the temporal structure of video.
Marengo 3.5를 출시했습니다. 비디오의 시간 구조를 중심으로 만든 비디오 네이티브 멀티모달 임베딩 모델입니다.
TwelveLabs launches Pegasus 1.5, a video-language model for long-form video. In TwelveLabs’ own comparison, it outperformed Gemini 3.1 Pro on segmentation and multimodal prompting.
TwelveLabs가 롱폼 비디오를 위한 비디오-언어 모델 Pegasus 1.5를 출시했습니다. TwelveLabs가 자체 비교한 결과, 세그멘테이션과 멀티모달 프롬프팅에서 Gemini 3.1 Pro를 앞섰습니다.
TwelveLabs releases Marengo 3.0, a new standard for foundation models that understand the world in all its complexity.
TwelveLabs가 Marengo 3.0을 출시했습니다. 세상의 모든 복잡함을 이해하는 파운데이션 모델의 새로운 기준입니다.