• [2025-03] Our RL Agent training framework RAGEN is online. Github ⭐️ 1.7k
• [2025-03] Will organize the CVPR 2025 Workshop on Foundation Models for Embodied Agents, with Ruohan Zhang, Yunzhu Li, Jiayuan Mao and Wenlong Huang, Qineng Wang, Yonatan Bisk, Shenlong Wang, Fei-Fei Li and Jiajun Wu. Calling for papers!
• [2024-12] Will organize the tutorial on Foundation Models for Embodied Agents at AAAI 2025 and NAACL 2025, with Yunzhu Li, Jiayuan Mao and Wenlong Huang.
• [2024-12] Selected as AAAI 2025 New Faculty Highlights.
• [2024-12] We released HourVideo, consists of a novel task suite comprising summarization, perception (recall, tracking), visual reasoning (spatial, temporal, predictive, causal, counterfactual), and navigation (room-to-room, object retrieval) tasks.
• [2024-11] Organized EMNLP 2024 Bird of Feather on LLMs for Embodied Agents with Joyce Chai, Parisa Kordjamshidi, Shuyan Zhou, Xudong Lin.
• [2024-11] Our Embodied Agent Interface has been selected as Best Paper Award (top 0.4%) at SoCal NLP 2024!
• [2024-10] Our Embodied Agent Interface has been selected as Oral Presentation (top 0.6% over D&B track, top 0.4% over all tracks) at NeurIPS 2024!
• [2024-10] We released Embodied Agent Interface with 100+ page analysis and built up symbolic simulators on BEHAVIOR.
• [2024-08] ACL 2024 Outstanding Paper on LM-Steer. A big shoutout to Chi Han, a star student awarded NAACL 2024 Outstanding Paper and ACL 2024 Outstanding Paper, and will be on the faculty job market!
• [2024-08] Serve as NAACL 2025 Organizing Committee (Publication Chairs).
• [2024-06] Serve as ACL 2025 Organizing Committee (Virtual Infrastructure Chairs).
• [2024-02] Serve as EMNLP 2024 Organizing Committee (Demonstration Chairs).
• [2023-08] Joined Stanford Vision and Learning lab as a postdoc working with Prof Jiajun Wu. Excited!
• [2023-05] Will join Northwestern Computer Science as a tenure-track assistant professor.
• [2022-10] Will present at EE CS Rising Star workshop in UT Austin on Oct 25-26 as a Riser.
• [2022-10] Will present at DARPA Forward event on Oct 4-5 as a DARPA Riser.
• [2022-07] Will give a tutorial at NAACL T3: New Frontiers of Information Extraction at 9:00–12:30 on July 10th (Location: Columbia D) with Muhao Chen, Lifu Huang, Ben Zhou, Heng Ji and Dan Roth. Welcome to our tutorial!

• [2022-03] Our work on COVID-KG is reported by the department Illinois CS Papers Earn Top Honors at Conferences.
• [2022-03] I am nominated as a DARPA Riser by DARPA.
• [2021-10] I am thrilled to receive Microsoft Research PhD Fellowship! Thanks Microsoft!
• [2021-08] Did a tutorial at ACL 2021 about Event-Centric Natural Language Processing with Muhao Chen, Qiang Ning, Hongming Wang, Heng Ji, Kathleen McKeown and Dan Roth.
• [2021-06] Our COVID-KG was awarded as Best Demo Paper at NAACL 2021.
• [2021-06] Our work of knowledge extraction and reasoning has been reported by the department Manling Li Produces Award-winning Event Schema Research with Advisor and Mentor Heng Ji.
• [2021-05] Awarded C.L. and Jane Liu Award for showing exceptional research promise.
• [2021-04] Selected as Mavis Future Faculty Fellow.
• [2021-02] Did a tutorial at AAAI 2021 about Event-centric Natural Language Understanding with Muhao Chen, Qiang Ning, Hongming Wang, Heng Ji and Dan Roth.
• [2020-10] Ranked 1st in TAC Streaming Multimedia Knowledge Base Population (SM-KBP) 2020.
• [2020-08] Our GAIA multimedia IE system has been awarded as Best Demo Paper at ACL 2020.
• [2019-06] Invited to present our event-tracking demo at DARPA Demo Day and ARL Transition Day.
• …

Please find my latest publications on Google Scholar.

2025

RAGEN (RL-Agent): Training Agents by Reinforcing Reasoning [Website][PDF][Code][Experimental Logs][td;lr]
Zihan Wang*, Kangrui Wang*, Qineng Wang*, Pingyue Zhang*, Linjie Li*, Zhengyuan Yang, Kefan Yu, Minh Nhat Nguyen, Monica Lam, Yiping Lu, Kyunghyun Cho, Jiajun Wu, Li Fei-Fei, Lijuan Wang, Yejin Choi, Manling Li
Open-source Project: LLM Agent Training via Reinforcement Learning, featuring:
- RL training for LLM Agents under a MDP formulation
- Clean, Minimized Code Organization
- Multi-Env, Multi-Action
- Secret Sauce for RL Training algorithms
- Easy-to-use Agent Environment Protocol

VAGEN: Training VLM Agents with Multi-Turn Reinforcement Learning [Blog][Code][td;lr]
Kangrui Wang*, Pingyue Zhang*, Zihan Wang*, Qineng Wang*, Linjie Li*, Zhengyuan Yang, Chi Wan, Yiping Lu, Manling Li
Open-source Project: VAGEN is a multi-turn reinforcement learning framework designed specifically for training VLM Agents. VAGEN leverages the TRICO algorithm to efficiently train VLMs for visual agentic tasks.

Chain-of-Experts: Unlocking the Communication Power of MoEs [Blog][Code][Tweet]
Zihan Wang, Rui Pan, Lu Yin, Manling Li, Shiwei Liu
Open-source Project: CoE achieves 17.6%-42% lower memory usage and reduces Math task validation loss from 1.20 to 1.12 with comparable compute.

T*: Re-thinking Temporal Search for Long-Form Video Understanding [Website][PDF][Data][Code]
Jinhui Ye*, Zihan Wang*, Haosen Sun, Keshigeyan Chandrasegaran, Zane Durante, Cristobal Eyzaguirre, Yonatan Bisk, Juan Carlos Niebles, Ehsan Adeli, Li Fei-Fei, Jiajun Wu, Manling Li
CVPR 2025

LayoutVLM: Differentiable Optimization of 3D Layout via Vision-Language Models [Website][PDF][Code]
Fan-Yun Sun, Weiyu Liu, Siyi Gu, Dylan Lim, Goutam Bhat, Federico Tombari, Manling Li, Nick Haber, Jiajun Wu
CVPR 2025

Why Is Spatial Reasoning Hard for VLMs? An Attention Mechanism Perspective on Focus Areas [PDF][Code][Data]
Shiqi Chen, Tongyao Zhu, Ruochen Zhou, Jinghan Zhang, Siyang Gao, Juan Carlos Niebles, Mor Geva, Junxian He, Jiajun Wu, Manling Li
ICML 2025

Bring Reason to Vision: Understanding Perception and Reasoning through Model Merging [Website][PDF][Code][Data]
Shiqi Chen, Jinghan Zhang, Tongyao Zhu, Wei Liu, Siyang Gao, Miao Xiong, Manling Li, Junxian He
ICML 2025

SyncMind: Measuring Agent Out-of-Sync Recovery in Collaborative Software Engineering [Website][PDF][Code][Data]
Xuehang Guo, Xingyao Wang, Yangyi Chen, Sha Li, Chi Han, Manling Li, Heng Ji
ICML 2025

EmbodiedBench: Comprehensive Benchmarking Multi-modal Large Language Models for Vision-Driven Embodied Agents [Website][PDF][Code]
Rui Yang, Hanyang Chen, Junyu Zhang, Mark Zhao, Cheng Qian, Kangrui Wang, Qineng Wang, Teja Venkat Koripella, Marziyeh Movahedi, Manling Li, Heng Ji, Huan Zhang, Tong Zhang
ICML 2025

Foundation Models Meet Embodied Agents [Website/Slides/Videos]
Manling Li, Yunzhu Li, Jiayuan Mao, Wenlong Huang
AAAI 2025: Tutorial
NAACL 2025: Tutorial

Chain-of-Action: Faithful and Multimodal Question Answering through Large Language Models [PDF]
Zhenyu Pan, Haozheng Luo, Manling Li, Han Liu
ICLR 2025

2024

Embodied Agent Interface: Benchmarking LLMs for Embodied Decision Making [Website][PDF][Code][Data][Docker][PyPi][Doc]
Manling Li^*, Shiyu Zhao^*, Qineng Wang^*, Kangrui Wang^*, Yu Zhou^*, Sanjana Srivastava, Cem Gokmen, Tony Lee, Li Erran Li, Ruohan Zhang, Weiyu Liu, Percy Liang, Li Fei-Fei, Jiayuan Mao, Jiajun Wu
NeurIPS 2024 Benchmark Track (Oral, Top 0.6%)
Best Paper Award at SoCal NLP 2024, Top 0.4%

HourVideo: 1-Hour Video-Language Understanding [Website][PDF][Data][Code]
Keshigeyan Chandrasegaran, Agrim Gupta, Taran Kota, Lea M. Hadzic, Jimming He, Cristobal Eyzaguirre, Zane Durante, Manling Li, Jiajun Wu, Li Fei-Fei
NeurIPS 2024 Benchmark Track

IKEA Manuals at Work: 4D Grounding of Assembly Instructions on Internet Videos [Website][PDF][Data][Code]
Yunong Liu, Weiyu Liu, Shubh Khanna, Cristobal Eyzaguirre, Manling Li, Juan Carlos Niebles, Vineeth Ravi, Saumitra Mishra, Jiajun Wu
NeurIPS 2024 Benchmark Track

Visually Descriptive Language Modeling for Vector Graphics Reasoning [PDF][Website][Code]
Zhenhailong Wang, Joy Hsu, Xingyao Wang, Kuan-Hao Huang, Manling Li, Jiajun Wu, Heng Ji
arXiv

LM-Steer: Word Embeddings Are Steers for Language Models [Website][PDF][Code][Live Demo][Slides][Poster]
Chi Han, Jialiang Xu, Manling Li, Yi Fung, Chenkai Sun, Nan Jiang, Tarek Abdelzaher, Heng Ji
ACL 2024
(Outstanding Paper Award at ACL 2024)

Why Does New Knowledge Create Messy Ripple Effects in LLMs? [PDF]
Jiaxin Qin, Zixuan Zhang, Chi Han, Pengfei Yu, Manling Li, Heng Ji
EMNLP 2024

Deep Concept Injection for Zero-shot Multimodal Reasoning [PDF]
Xudong Lin, Manling Li, Richard Zemel, Heng Ji, Shih-Fu Chang
EMNLP 2024

Knowledge Overshadowing Causes Amalgamated Hallucination in Large Language Models [PDF]
Yuji Zhang, Sha Li, Jiateng Liu, Pengfei Yu, Yi Fung, Jing Li, Manling Li, Heng Ji
arXiv

MentalArena: Self-play Training of Language Models for Diagnosis and Treatment of Mental Health Disorders [PDF]
Cheng Li, May Fung, Qingyun Wang, Chi Han, Manling Li, Jindong Wang, Heng Ji
arXiv

Can LLMs Produce Faithful Explanations For Fact-checking? Towards Faithful Explainable Fact-Checking via Multi-Agent Debate [PDF]
Kyungha Kim*, Sangyun Lee*, Kung-Hsiang Huang*, Hou Pong Chan, Manling Li, Heng Ji
arXiv

InfoPattern: Unveiling Information Propagation Patterns in Social Media [PDF]
Chi Han*, Jialiang Xu*, Manling Li* , Hanning Zhang*, Tarek Abdelzaher, Heng Ji
arXiv

SmartBook: AI-Assisted Situation Report Generation [PDF]
Revanth Gangi Reddy, Yi Fung, Qi Zeng, Manling Li, Zihan Wang, Paul Sullivan, Heng Ji
arXiv

Controlling Object Existence Hallucinations in Large Vision Language Models [PDF]
Bohan Zhai, Shijia Yang, Chenfeng Xu, Sheng Shen, Kurt Keutzer, Chunyuan Li, Manling Li
arXiv

2023

Event-centric Multimodal Knowledge Acquisition [PDF]
Manling Li
Thesis Committee: Heng Ji, Jiawei Han, Chengxiang Zhai, Shih-Fu Chang, Kyunghyun Cho
Thesis

ViStruct: Visual Structural Knowledge Extraction via Curriculum Guided Code-Vision Representation [PDF]
Yangyi Chen, Xingyao Wang, Manling Li, Derek Hoiem, Heng Ji
EMNLP 2023

Defining a New NLP Playground [PDF]
Sha Li, Chi Han, Pengfei Yu, Carl Edwards, Manling Li, Xingyao Wang, Yi Fung, Charles Yu, Joel R. Tetreault, Eduard H Hovy, Heng Ji
EMNLP 2023 Findings

Knowledge-Driven Vision-Language Encoding [Website]
Manling Li, Xudong Lin, Jie Lei, Mohit Bansal, Carl Vondrick, Shih-Fu Chang, Heng Ji
CVPR 2023: Tutorial

Towards Fast Adaptation of Pretrained Contrastive Models for Multi-channel Video-Language Retrieval [PDF] [Code]
Xudong Lin, Simran Tiwari, Shiyuan Huang, Manling Li, Mike Zheng Shou, Heng Ji, Shih-Fu Chang
CVPR 2023

Open Visual Knowledge Extraction via Relation-Oriented Multimodality Model Prompting [Website]
Hejie Cui, Xinyu Fang, Zihan Zhang, Ran Xu, Xuan Kan, Xin Liu, Manling Li, Yangqiu Song, Carl Yang
NeurIPS 2023

Non-Sequential Graph Script Induction via Multimedia Grounding [PDF]
Yu Zhou^†, Sha Li, Manling Li, Xudong Lin, Shih-Fu Chang, Mohit Bansal and Heng Ji
ACL 2023 (^† denotes supervised undergraduate)

A Language First Approach to Procedure Planning [PDF]
Jiateng Liu^†, Sha Li, Zhenhailong Wang, Manling Li, Heng Ji
ACL 2023 Findings (^† denotes supervised undergraduate)

Open-Domain Hierarchical Event Schema Induction by Incremental Prompting and Verification [PDF]
Sha Li, Ruining Zhao^†, Manling Li, Heng Ji, Chris Callison-Burch and Jiawei Han
ACL 2023 (^† denotes supervised undergraduate)

Multimedia Generative Script Learning for Task Planning [PDF]
Qingyun Wang, Manling Li, Hou Pong Chan, Lifu Huang, Julia Hockenmaier, Girish Chowdhary and Heng Ji
ACL 2023 Findings

Learning to Decompose Visual Features with Latent Textual Prompts [PDF] [Code]
Feng Wang^†, Manling Li, Xudong Lin, Hairong Lv, Alexander Schwing, Heng Ji
ICLR 2023 (^† denotes supervised undergraduate)

Knowledge-Driven Vision-Language Pretraining [PDF] [Website]
Manling Li, Xudong Lin, Jie Lei, Mohit Bansal, Shih-Fu Chang, Heng Ji
AAAI 2023: Tutorial

Video Event Extraction via Tracking Visual States of Arguments [PDF] [Code]
Guang Yang^†, Manling Li, Jiajie Zhang, Xudong Lin, Shih-Fu Chang, Heng Ji
AAAI 2023 (^† denotes supervised undergraduate)

ADEPT: A DEbiasing PrompT Framework [PDF] [Code]
Ke Yang^†, Charles Yu, Yi Fung, Manling Li, Heng Ji
AAAI 2023 (^† denotes supervised undergraduate)

2022

Language Models with Image Descriptors are Strong Few-Shot Video-Language Learners [PDF] [Code]
Zhenhailong Wang^†*,Manling Li*, Ruochen Xu, Luowei Zhou, Jie Lei, Xudong Lin, Shuohang Wang, Ziyi Yang, Chenguang Zhu, Derek Hoiem, Shih-Fu Chang, Mohit Bansal, Heng Ji
NeurIPS'22 (equal contribution)

CLIP-Event:Connecting Vision and Text with Event Structures [PDF] [Data] [Code]
Manling Li, Ruochen Xu, Shuohang Wang, Xudong Lin, Chenguang Zhu, Xuedong Huang, Heng Ji, Shih-Fu Chang
CVPR'22 (Oral, Top 4.1%)

COVID-19 Claim Radar: A Structured Claim Extraction and Tracking System [PDF] [Code] [Demo] [Video]
Manling Li, Revanth Gangi Reddy, Ziqi Wang, Yi-Shyuan Chiang, Tuan M. Lai, Pengfei Yu, Zixuan Zhang,Heng Ji
ACL'22 Demo

Event Schema Induction with Double Graph Autoencoders [PDF] [Code]
Xiaomeng Jin^†, Manling Li and Heng Ji
NAACL'22

New Frontiers of Information Extraction [PDF] [Website] [Slides] [Videos]
Muhao Chen, Lifu Huang, Manling Li, Ben Zhou, Heng Ji
NAACL'22: Tutorial

MuMuQA: Multimedia Multi-Hop News Question Answering via Cross-Media Knowledge Extraction and Grounding [PDF] [Data]
Revanth Gangi Reddy^†, Xilin Rui^†, Manling Li, Xudong Lin, Haoyang Wen, Jaemin Cho, Lifu Huang, Mohit Bansal, Avi Sil, Shih-Fu Chang, Alexander Schwing, Heng Ji
AAAI'22

2021

The Future is not One-dimensional: Complex Event Schema Induction by Graph Modeling for Event Prediction [PDF] [Data]
Manling Li, Sha Li, Zhenhailong Wang, Lifu Huang, Kyunghyun Cho, Heng Ji, Jiawei Han and Clare Voss
EMNLP'21

Timeline Summarization based on Event Graph Compression via Time-Aware Optimal Transport [PDF] [Data]
Manling Li, Tengfei Ma, Mo Yu, Lingfei Wu, Tian Gao, Heng Ji and Kathleen McKeown
EMNLP'21

Joint Multimedia Event Extraction from Video and Article [PDF] [Data]
Brian Chen, Xudong Lin, Christopher Thomas, Manling Li, Shoya Yoshida, Lovish Chum, Heng Ji and Shih-Fu Chang
EMNLP'21 Findings

Event-centric Natural Language Processing [PDF] [Slides]
Muhao Chen, Hongming Zhang, Qiang Ning, Manling Li, Heng Ji, Kathleen McKeown and Dan Roth
ACL'21: Tutorial.

COVID-19 Literature Knowledge Graph Construction and Drug Repurposing Report Generation [PDF] [Code/Data]
Qingyun Wang, Manling Li, Xuan Wang, Nikolaus Parulian, Guangxing Han, Jiawei Ma, Jingxuan Tu, Ying Lin, Haoran Zhang, Weili Liu, Aabhas Chauhan, Yingjun Guan, Bangzheng Li, Ruisong Li, Xiangchen Song, Heng Ji, Jiawei Han, Shih-Fu Chang, James Pustejovsky, David Liem, Ahmed Elsayed, Martha Palmer, Jasmine Rah, Clare Voss, Cynthia Schneider, Boyan Onyshkevych
NAACL'21: System Demonstrations
(Best Demo Paper Award at NAACL2021)

RESIN: A Dockerlized Schema-Guided Cross-document Cross-lingual Cross-media Information Extraction and Event Tracking System [PDF] [Code]
Haoyang Wen, Ying Lin, Tuan M. Lai, Xiaoman Pan, Sha Li, Xudong Lin, Ben Zhou, Manling Li, Haoyu Wang, Hongming Zhang, Xiaodong Yu, Alexander Dong, Zhenhailong Wang, Yi R. Fung, Piyush Mishra, Qing Lyu, Dídac Surís, Brian Chen, Susan W. Brown, Martha Palmer, Chris Callison-Burch, Carl Vondrick, Jiawei Han, Dan Roth, Shih-Fu Chang and Heng Ji
NAACL'21: System Demonstrations

Event-centric Natural Language Processing [PDF] [Slides]
Muhao Chen, Hongming Zhang, Qiang Ning, Manling Li, Heng Ji and Dan Roth
AAAI'21: Tutorial.

2020

Connecting the Dots: Event Graph Schema Induction with Path Language Modeling [PDF] [Code/Data] [Slides]
Manling Li, Qi Zeng, Ying Lin, Kyunghyun Cho, Heng Ji, Jonathan May, Nathanael Chambers and Clare Voss
EMNLP'20: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing.

GAIA: A Fine-grained Multimedia Knowledge Extraction System [PDF] [Code] [Video]
Manling Li*, Alireza Zareian*, Ying Lin, Xiaoman Pan, Spencer Whitehead, Brian Chen, Bo Wu, Heng Ji, Shih-Fu Chang, Clare R. Voss, Dan Napierski, Marjorie Freedman
ACL'20: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations. pp. 77–86
(Best Demo Paper Award at ACL2020)

Cross-media Structured Common Space for Multimedia Event Extraction [PDF] [Code] [Slides]
Manling Li*, Alireza Zareian*, Qi Zeng, Spencer Whitehead, Di Lu, Heng Ji, Shih-Fu Chang
ACL'20: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp.2557–2568

GAIA at SM-KBP 2020 - A Dockerized Multi-media Multi-lingual Knowledge Extraction, Clustering, Temporal Tracking and Hypothesis Generation System [PDF] [Project]
Manling Li, Ying Lin, Tuan Manh Lai, Xiaoman Pan, Haoyang Wen, Sha Li, etc %Zhenhailong Wang, Pengfei Yu, Lifu Huang, Di Lu, Qingyun Wang, Haoran Zhang, Qi Zeng, Chi Han, Zixuan Zhang, Yujia Qin, Xiaodan Hu, Nikolaus Parulian, Daniel Campos, Heng Ji, Brian Chen, Xudong Lin, Alireza Zareian, Amith Ananthram, Emily Allaway, Shih-Fu Chang, Kathleen McKeown, Yixiang Yao, Yifan Wang, Michael Spector, Mitchell DeHaven, Daniel Napierski, Marjorie Freedman, Pedro Szekely, Haidong Zhu, Ram Nevatia, Yang Bai, Yifan Wang, Ali Sadeghian, Haodi Ma, Daisy Zhe Wang
TAC-KBP: Text Analysis Conference Knowledge Base Population Workshop 2020 (Rank 1st in the leaderboard.)

2019

Keep Meeting Summaries on Topic: Abstractive Multi-Modal Meeting Summarization
[PDF] [Code]
Manling Li, Lingyu Zhang, Heng Ji, Rich Radke
ACL'19: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp.2190–2196

Multilingual Entity, Relation, Event and Human Value Extraction [PDF] [Code] [Video]
Manling Li, Ying Lin, Joe Hoover, Spencer Whitehead, Clare Voss, Morteza Dehghani, Heng Ji
NAACL'19: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (Demonstrations), pp.110–115

GAIA at SM-KBP 2019 - A Multi-media Multi-lingual KnowledgeExtraction and Hypothesis Generation System [PDF] [Project]
Manling Li, Ying Lin, Ananya Subburathinam, Spencer Whitehead, Xiaoman Pan, Di Lu, Qingyun Wang, Tongtao Zhang, Lifu Huang, Heng Ji, Alireza Zareian, Hassan Akbari, Brian Chen, Bo Wu, Emily Allaway, Shih-Fu Chang, Kathleen McKeown, Yixiang Yao, Jennifer Chen, Eric Berquist, Kexuan Sun, Xujun Peng, Ryan Gabbard Marjorie Freedman, Pedro Szekely, T.K. Satish Kumar, Arka Sadhu, Ram Nevatia, Miguel Rodriguez, Yifan Wang, Yang Bai, Ali Sadeghian, Daisy Zhe Wang
TAC-KBP: Text Analysis Conference Knowledge Base Population Workshop 2019 (Rank 1st, with more than 10% higher than the second team.)

2018 and Before

Please see my full list at Google Scholar.

Tutorials

• Foundation Models Meet Embodied Agents
  Tutorial at AAAI 2025. Feb, 2025.
  Tutorial at NAACL 2025. May, 2024.
  With Yunzhu Li, Jiayuan Mao, Wenlong Huang.
  [Website/Slides]
  

• Lifecycle of Knowledge in LLMs: Memorization, Editing, and Beyond
  Tutorial at AAAI 2025. Feb, 2025.
  With Yuji Zhang, Zoey Sha Li, Chi Han, Heng Ji.
  [Website/Slides]
  

• Beyond Human Creativity: A Tutorial on Advancements in AI Generated Content (AIGC)
  Tutorial at IJCAI 2024. Aug, 2024.
  With Bang Liu, Yu Chen, Heng Ji, Lingfei Wu.
  [Website/Slides]
  

• Large Language Models for NLP in Finance
  Tutorial at ICAIF (ACM International Conference on AI in Finance) 2023. Nov, 2023.
  With Simerjot Kaur, Charese Smiley, Xiaomo Liu, Elena Kochkina, Mohammad Ghassemi, Reza Khanmohammadi.
  [Slides]
  

• Knowledge-Driven Vision-Language Encoding
  Tutorial at CVPR 2023. Jun, 2023.
  With Xudong Lin, Jie Lei, Carl Vondrick, Mohit Bansal, Heng Ji, and Shih-Fu Chang.
  [Website/Slides/Videos] [ReadingList]
  

• Knowledge-Driven Vision-Language Pretraining
  Tutorial at AAAI 2023. Feb, 2023.
  With Xudong Lin, Jie Lei, Mohit Bansal, Heng Ji, and Shih-Fu Chang.
  [Website/Slides/Videos] [ReadingList]
  

• New Frontiers of Information Extraction
  Tutorial at NAACL 2022. Jul, 2022.
  With Muhao Chen, Lifu Huang, Ben Zhou, Heng Ji, and Dan Roth.
  [Website/Slides/Videos] [PDF]

• Event-centric Natural Language Understanding
  Tutorial at ACL2021. Aug 2021.
  With Muhao Chen, Qiang Ning, Hongming Zhang, Heng Ji, Kathleen McKeown, and Dan Roth.
  [Website/Slides/Videos] [PDF]

• Event-centric Natural Language Processing
  Tutorial at AAAI2021. Feb, 2021.
  With Muhao Chen, Qiang Ning, Hongming Zhang, Heng Ji, and Dan Roth.
  [Website/Slides/Videos] [PDF]

Invited Talks

• Agent Training Under a MDP Formulation
  AAAI 2025 New Faculty Highlights. Feb 2025
  LM4Plan @ AAAI 2025. Feb 2025
  

• Embodied Agent Interface: LLMs and VLMs for Embodied Reasoning and Planning
  SFU @ NeurIPS 2024. Dec 2024
  

• LLMs for Embodied Agents
  EMNLP 2024 Birds of Feather. Nov 2024
  

• Customizing Large Language Models to Embodied Agent interacting with Embodied Envioronments
  EMNLP 2024 CustomNLP4U Workshop. Nov 2024
  

• From Large Language Models to Large Agent Models
  Keynote at Amazon-Illinois Center on AI for Interactive Conversational Experiences Fall Research Symposium 2024. Sept 2024
  

• Reasoning and Planning with Physical World Knowledge
  2024 Allerton Conference on Communication, Control, and Computing. Sept 2024
  

• Embodied Agent Interface: LLMs for Embodied Decision Making
  TTIC Multimodal AI Workshop 2024. Aug 2024
  

• Multimodal Knowledge for Social Good
  Summer Institute in Computational Social Science 2024. Aug 2024
  

• Reasoning, Planning and Compositionality in Multimodality
  SpLU-RoboNLP 2024 Workshop at ACL. Jul 2024
  

• Visually Descriptive Language Modeling for Document Intelligence
  Adobe Research. Jul 2024
  

• From Large Language Models to Large Agent Models
  Apple NLU Workshop 2024. Jun 2024
  

• From Words to Worlds: A Close Look to Diffusion Models (through an NLP Lens)
  UIUC NLP Seminar. Jun 2024
  

• The Missing Knowledge in LLMs to Interact with the Physical World
  Midwest Machine Learning Symposium 2024. May 2024
  

• Beyond the Beaten Path: Exploring the Role of Graphs in Multimodal Foundation Models
  Keynote Talk at NeurIPS 2023 Workshop on New Frontiers in Graph Learning. Dec 2023
  

• LLMs for robotics: Modeling the Knowledge of the Physical World
  Stanford Vision and Learning Seminar. Oct 2023
  

• Knowledge Foundation Models
  Adobe Research. Oct 2023
  

• Modeling the Semantics of the Physical World
  Stanford CogAI. Jun 2023
  

• Towards Factuality in Information Access: Multimodal Knowledge Acqusition and Reasoning [Slides]
  Carnegie Mellon University, LTI. Feb 2023
  Northwestern University, CS. Feb 2023
  Northeastern University, ECE. Feb 2023
  Purdue University, CS. Feb 2023
  Rice University, CS. Feb 2023
  Virginia Tech, CS. Feb 2023
  Max Planck Institute. Feb 2023
  UVA, CS. Mar 2023
  MBZUAI. Mar 2023
  U Washington St Louis, CS. Mar 2023
  University of Toronto, CS+ECE. Mar 2023
  UC San Diego, ECE. Feb 2023
  UC Davis, CS. Mar 2023
  UC Los Angeles, ECE. Apr 2023
  

• From Entity-Centric to Event-Centric Multimodal Event Knowledge Acquisition
  EE CS Rising Star, University of Texas at Austin, USA. Oct 2022.
  

• Towards Accurate Intelligent Analysis: Event-Centric Multimedia Knowledge Extraction
  DARPA Forward (Invite-Only), USA. Oct 2022.

• Event-Centric Multimedia Data Understanding
  Ohio State University, USA. Oct 2022.
  Singapore Management University, Singapore. Oct 2022.
  George Mason University, USA. Oct 2022.
  North Carolina State University, USA. Oct 2022.
  

• Multimedia Event Extraction: From Object-Centric to Event-Centric
  Virginia Tech, USA. Sept 2022.

• Event Knowledge Graph Construction
  LOGS Graph Reasoning Seminar. Aug 2022.

• Event Graph Structures in Vision-Language Understanding
  DataFun. Jun 2022.
  

• Connecting Vision and Text using Event Structures
  NewsBreak. Apr 2022.
  

• Memories as Repositories of Events: Structural Event Knowledge Acquisition
  University of Notre Dame. Feb 2022.
  

• Comprehensive Event Understanding in Multimedia Data
  USC ISI. Dec 2021.
  

• Structural Event Knowledge Acquisition from Multimedia Data
  UIUC NLP Seminar. Nov 2021.
  

• Event Extraction and Reasoning in Multimedia News Data
  Microsoft Research. Nov 2021.
  

• Improving Visual Event and Argument Role Understanding with Contrastive Image-Language Pretraining
  Microsoft Research. Aug 2021.
  

• Fine-Grained Knowledge Extraction System from Multimedia Data
  ai.science. Oct 2020.
  

• Event Understanding and Narration for Multimedia Data
  Intel MDI Research Lab. May 2020.
  

Organizing Committee:

• ACL 2025 (Virtual Infrastructure Chairs)
• NAACL 2025 (Publication Chairs) summary
• EMNLP 2024 (Demonstration Chairs) summary
• Foundation Models Meet Embodied Agents @ CVPR 2025 workshop
• Towords Knowledgeable Foundation Models @ ACL 2025 workshop
• Towords Knowledgeable Foundation Models @ AAAI 2025 workshop
• Towords Knowledgeable Foundation Models @ ACL 2024 workshop
• Knowledge Discovery from Unstructured Data in Financial Services (KDF) @ SIGIR 2023 Workshop

Senior Area Chair:

• EMNLP (from 2025)

Area Chair:

• ACL (from 2023)
• EMNLP (from 2023)
• NAACL (from 2024)
• COLM (from 2025)

Program Committee:

• ARR (ACL Rolling Review, from 2021)
• ACL (from 2021)
• EMNLP (from 2021)
• NAACL-HLT (from 2021)
• AAAI (from 2021)
• WWW (from 2021)
• AKBC (2021)
• EACL (from 2021)
• KDD DI Workshop (2021)
• NLPCC (2021)
• COLING (from 2020)
• AACL (from 2020)
• CCL (from 2020)

Journal Reviewer:

• TACL
• TIST
• TKDE

Community Services:

• ACL Student Research Workshop (SRW) Mentor, ACL, 2024
• ACM Mentor, ACM Mentorship Program at UIUC, 2022
• CS Ambassador, UIUC CS Visit Day for Prospective Graduate Students, 2022
• Advising Assistant, UIUC PhD Orientation Seminar, 2021
• Graduate Student Representative, UIUC CS Visit Day, 2020

Awards

ACL 2024 Outstanding Paper Award
SoCal NLP 2024 Best Paper Award
NAACL 2021 Best Demo Paper Award
ACL 2020 Best Demo Paper Award
AAAI 2025 New Faculty Highlights
Microsoft Research Postdoc Fellowship 2023
EE CS Rising Star 2022
DARPA Riser 2022
Microsoft Research PhD Fellowship 2021
Mavis Future Faculty Fellow
C.L. and Jane Liu Award
National Scholarship

Academic and Scientific Competitions

Ranked 1st in NIST TAC Streaming Multimedia Knowledge Base Population (SM-KBP) 2020
Ranked 1st in NIST TAC Streaming Multimedia Knowledge Base Population (SM-KBP) 2019, with more than 10% absolute gains compared the second ranked team

Instructor:

• COMP_SCI 396 Reasoning and Planning in the Foundation Model Era
  NU, Winter 2025
• COMP_SCI 496 Agentic AI
  NU, Spring 2025

Guest Lecturer:

• Navigating Up and Down in the Job Searching
  COMP_SCI 496 Academic Job Search
  NU, Fall 2023
• Event-Centric Multimedia Encoding
  CS 6604 Advanced Topics in Natural Language Processing
  Virginia Tech, Fall 2022
• Knowledge-Driven Vision-Language Pretraining
  CS 546 Advanced Topics in Natural Language Processing
  UIUC, Fall 2022
• Recent Advances in Multimedia Encoding
  CS 546 Advanced Topics in Natural Language Processing
  UIUC, Fall 2022
• Timeline Summarization: Introducing Temporal Dimensions into Summarization
  CS 598 Knowledge Driven Natural Language Generation
  UIUC, Spring 2022
• Multimedia Encoding via Vision-Language Pretraining
  CS 546 Advanced Topics in Natural Language Processing
  UIUC, Fall 2021