• [2024-02] Serve as EMNLP 2024 Organizing Committee (Demonstration Chairs).
• [2024-02] Got my very first PhD student Zihan! Super excited! Thanks for the trust and I will do my best to be the best advisor like Heng.
• [2023-12] Keynote Talk “Beyond the Beaten Path: Exploring the Role of Graphs in Multimodal Foundation Models” at NeurIPS 2023 Workshop on New Frontiers in Graph Learning. Panel discussion with Jiawei Han, Marinka Zitnik, Yanqi Zhou, Mohammad Shoeybi.
• [2023-10] Will organize “Towards Knowledgeable Language Models @ ACL24 Workshop” Call for Papers, submission deadline in May 20, 2024.
• [2023-08] Joined Stanford Vision and Learning lab as a postdoc working with Prof Jiajun Wu. Excited!
• [2023-07] Served as a session chair for multimodality at ACL 2023.
• [2023-06] Will serve as an Area Chair at EMNLP 2023.
• [2023-05] Will join Northwestern Computer Science as a tenure-track assistant professor.
• [2022-12] Will serve as an Area Chair at ACL 2023.
• [2022-10] Will present at EE CS Rising Star workshop in UT Austin on Oct 25-26 as a Riser.
• [2022-10] Will present at DARPA Forward event in OSU on Oct 4-5 as a DARPA Riser.
• [2022-07] Will give a tutorial at NAACL T3: New Frontiers of Information Extraction at 9:00–12:30 on July 10th (Location: Columbia D) with Muhao Chen, Lifu Huang, Ben Zhou, Heng Ji and Dan Roth. Welcome to our tutorial!

• [2022-03] Our work on COVID-KG is reported by the department Illinois CS Papers Earn Top Honors at Conferences.
• [2022-03] I am nominated as a DARPA Riser by DARPA.
• [2021-10] I am thrilled to receive Microsoft Research PhD Fellowship! Thanks Microsoft!
• [2021-08] Did a tutorial at ACL 2021 about Event-Centric Natural Language Processing with Muhao Chen, Qiang Ning, Hongming Wang, Heng Ji, Kathleen McKeown and Dan Roth.
• [2021-06] Our COVID-KG was awarded as Best Demo Paper at NAACL 2021.
• [2021-06] Our work of knowledge extraction and reasoning has been reported by the department Manling Li Produces Award-winning Event Schema Research with Advisor and Mentor Heng Ji.
• [2021-05] Awarded C.L. and Jane Liu Award for showing exceptional research promise.
• [2021-04] Selected as Mavis Future Faculty Fellow.
• [2021-02] Did a tutorial at AAAI 2021 about Event-centric Natural Language Understanding with Muhao Chen, Qiang Ning, Hongming Wang, Heng Ji and Dan Roth.
• [2020-10] Ranked 1st in TAC Streaming Multimedia Knowledge Base Population (SM-KBP) 2020.
• [2020-08] Our GAIA multimedia IE system has been awarded as Best Demo Paper at ACL 2020.
• [2019-06] Invited to present our event-tracking demo at DARPA Demo Day and ARL Transition Day.
• …

Books

Advances in Information Access
Sha Li, Manling Li, Heng Ji
Ongoing (World Scientifc Publishing)

Natural Language Processing: A Modern Perspective
Bang Liu, Yu Chen, Manling Li, Heng Ji, Lingfei Wu
Ongoing (Cambridge University Press)

Papers (* denotes equal contribution, ^† denotes mentored students)

2024

Embodied Agent Interface: Benchmarking LLMs for Embodied Decision Making [Website]
Manling Li^*, Shiyu Zhao^*, Qineng Wang^*, Kangrui Wang^*, Yu Zhou^*, Sanjana Srivastava, Cem Gokmen, Tony Lee, Li Erran Li, Ruohan Zhang, Weiyu Liu, Percy Liang, Li Fei-Fei, Jiayuan Mao, Jiajun Wu

HourVideo: 1-Hour Video-Language Understanding
Keshigeyan Chandrasegaran, Agrim Gupta, Taran Kota, Lea M. Hadzic, Jimming He, Cristobal Eyzaguirre, Zane Durante, Manling Li, Jiajun Wu, Li Fei-Fei

IKEA Manuals at Work: 4D Grounding of Assembly Instructions on Internet Videos [Website]
Yunong Liu, Weiyu Liu, Shubh Khanna, Cristobal Eyzaguirre, Manling Li, Juan Carlos Niebles, Vineeth Ravi, Saumitra Mishra, Jiajun Wu

Text-based Reasoning About Vector Graphics [PDF][Website][Code]
Zhenhailong Wang, Joy Hsu, Xingyao Wang, Kuan-Hao Huang, Manling Li, Jiajun Wu, Heng Ji
arXiv

Can LLMs Produce Faithful Explanations For Fact-checking? Towards Faithful Explainable Fact-Checking via Multi-Agent Debate [PDF]
Kyungha Kim*, Sangyun Lee*, Kung-Hsiang Huang *, Hou Pong Chan, Manling Li, Heng Ji
arXiv

Controlling Object Existence Hallucinations in Large Vision Language Models [PDF]
Bohan Zhai, Shijia Yang, Chenfeng Xu, Sheng Shen, Kurt Keutzer, Chunyuan Li, Manling Li
arXiv

LM-Steer: Word Embeddings Are Steers for Language Models [PDF]
Chi Han, Jialiang Xu, Manling Li, Yi Fung, Chenkai Sun, Nan Jiang, Tarek Abdelzaher, Heng Ji
ACL 2024

2023

Event-centric Multimodal Knowledge Acquisition [PDF]
Manling Li
Thesis Committee: Heng Ji, Jiawei Han, Chengxiang Zhai, Shih-Fu Chang, Kyunghyun Cho
Thesis

Deep Concept Injection for Zero-shot Multimodal Reasoning [PDF]
Xudong Lin, Manling Li, Richard Zemel, Heng Ji, Shih-Fu Chang
arXiv

ViStruct: Visual Structural Knowledge Extraction via Curriculum Guided Code-Vision Representation [PDF]
Yangyi Chen, Xingyao Wang, Manling Li, Derek Hoiem, Heng Ji
EMNLP 2023

Defining a New NLP Playground [PDF]
Sha Li, Chi Han, Pengfei Yu, Carl Edwards, Manling Li, Xingyao Wang, Yi Fung, Charles Yu, Joel R. Tetreault, Eduard H Hovy, Heng Ji
EMNLP 2023 Findings

Knowledge-Driven Vision-Language Encoding [Website]
Manling Li, Xudong Lin, Jie Lei, Mohit Bansal, Carl Vondrick, Shih-Fu Chang, Heng Ji
CVPR 2023: Tutorial

Towards Fast Adaptation of Pretrained Contrastive Models for Multi-channel Video-Language Retrieval [PDF] [Code]
Xudong Lin, Simran Tiwari, Shiyuan Huang, Manling Li, Mike Zheng Shou, Heng Ji, Shih-Fu Chang
CVPR 2023

Open Visual Knowledge Extraction via Relation-Oriented Multimodality Model Prompting [Website]
Hejie Cui, Xinyu Fang, Zihan Zhang, Ran Xu, Xuan Kan, Xin Liu, Manling Li, Yangqiu Song, Carl Yang
NeurIPS 2023

Non-Sequential Graph Script Induction via Multimedia Grounding [PDF]
Yu Zhou^†, Sha Li, Manling Li, Xudong Lin, Shih-Fu Chang, Mohit Bansal and Heng Ji
ACL 2023 (^† denotes supervised undergraduate)

A Language First Approach to Procedure Planning [PDF]
Jiateng Liu^†, Sha Li, Zhenhailong Wang, Manling Li, Heng Ji
ACL 2023 Findings (^† denotes supervised undergraduate)

Open-Domain Hierarchical Event Schema Induction by Incremental Prompting and Verification [PDF]
Sha Li, Ruining Zhao^†, Manling Li, Heng Ji, Chris Callison-Burch and Jiawei Han
ACL 2023 (^† denotes supervised undergraduate)

Multimedia Generative Script Learning for Task Planning [PDF]
Qingyun Wang, Manling Li, Hou Pong Chan, Lifu Huang, Julia Hockenmaier, Girish Chowdhary and Heng Ji
ACL 2023 Findings

Learning to Decompose Visual Features with Latent Textual Prompts [PDF] [Code]
Feng Wang^†, Manling Li, Xudong Lin, Hairong Lv, Alexander Schwing, Heng Ji
ICLR 2023 (^† denotes supervised undergraduate)

Knowledge-Driven Vision-Language Pretraining [PDF] [Website]
Manling Li, Xudong Lin, Jie Lei, Mohit Bansal, Shih-Fu Chang, Heng Ji
AAAI 2023: Tutorial

Video Event Extraction via Tracking Visual States of Arguments [PDF] [Code]
Guang Yang^†, Manling Li, Jiajie Zhang, Xudong Lin, Shih-Fu Chang, Heng Ji
AAAI 2023 (^† denotes supervised undergraduate)

ADEPT: A DEbiasing PrompT Framework [PDF] [Code]
Ke Yang^†, Charles Yu, Yi Fung, Manling Li, Heng Ji
AAAI 2023 (^† denotes supervised undergraduate)

2022

Language Models with Image Descriptors are Strong Few-Shot Video-Language Learners [PDF] [Code]
Zhenhailong Wang^†*,Manling Li*, Ruochen Xu, Luowei Zhou, Jie Lei, Xudong Lin, Shuohang Wang, Ziyi Yang, Chenguang Zhu, Derek Hoiem, Shih-Fu Chang, Mohit Bansal, Heng Ji
NeurIPS'22 (equal contribution)

CLIP-Event:Connecting Vision and Text with Event Structures [PDF] [Data] [Code]
Manling Li, Ruochen Xu, Shuohang Wang, Xudong Lin, Chenguang Zhu, Xuedong Huang, Heng Ji, Shih-Fu Chang
CVPR'22 (Oral)

COVID-19 Claim Radar: A Structured Claim Extraction and Tracking System [PDF] [Code] [Demo] [Video]
Manling Li, Revanth Gangi Reddy, Ziqi Wang, Yi-Shyuan Chiang, Tuan M. Lai, Pengfei Yu, Zixuan Zhang,Heng Ji
ACL'22 Demo

Event Schema Induction with Double Graph Autoencoders [PDF] [Code]
Xiaomeng Jin^†, Manling Li and Heng Ji
NAACL'22

New Frontiers of Information Extraction [PDF] [Website] [Slides] [Videos]
Muhao Chen, Lifu Huang, Manling Li, Ben Zhou, Heng Ji
NAACL'22: Tutorial

MuMuQA: Multimedia Multi-Hop News Question Answering via Cross-Media Knowledge Extraction and Grounding [PDF] [Data]
Revanth Gangi Reddy^†, Xilin Rui^†, Manling Li, Xudong Lin, Haoyang Wen, Jaemin Cho, Lifu Huang, Mohit Bansal, Avi Sil, Shih-Fu Chang, Alexander Schwing, Heng Ji
AAAI'22

2021

The Future is not One-dimensional: Complex Event Schema Induction by Graph Modeling for Event Prediction [PDF] [Data]
Manling Li, Sha Li, Zhenhailong Wang, Lifu Huang, Kyunghyun Cho, Heng Ji, Jiawei Han and Clare Voss
EMNLP'21

Timeline Summarization based on Event Graph Compression via Time-Aware Optimal Transport [PDF] [Data]
Manling Li, Tengfei Ma, Mo Yu, Lingfei Wu, Tian Gao, Heng Ji and Kathleen McKeown
EMNLP'21

Joint Multimedia Event Extraction from Video and Article [PDF] [Data]
Brian Chen, Xudong Lin, Christopher Thomas, Manling Li, Shoya Yoshida, Lovish Chum, Heng Ji and Shih-Fu Chang
EMNLP'21 Findings

Event-centric Natural Language Processing [PDF] [Slides]
Muhao Chen, Hongming Zhang, Qiang Ning, Manling Li, Heng Ji, Kathleen McKeown and Dan Roth
ACL'21: Tutorial.

COVID-19 Literature Knowledge Graph Construction and Drug Repurposing Report Generation [PDF] [Code/Data]
Qingyun Wang, Manling Li, Xuan Wang, Nikolaus Parulian, Guangxing Han, Jiawei Ma, Jingxuan Tu, Ying Lin, Haoran Zhang, Weili Liu, Aabhas Chauhan, Yingjun Guan, Bangzheng Li, Ruisong Li, Xiangchen Song, Heng Ji, Jiawei Han, Shih-Fu Chang, James Pustejovsky, David Liem, Ahmed Elsayed, Martha Palmer, Jasmine Rah, Clare Voss, Cynthia Schneider, Boyan Onyshkevych
NAACL'21: System Demonstrations
(Best Demo Paper Award at NAACL2021)

RESIN: A Dockerlized Schema-Guided Cross-document Cross-lingual Cross-media Information Extraction and Event Tracking System [PDF] [Code]
Haoyang Wen, Ying Lin, Tuan M. Lai, Xiaoman Pan, Sha Li, Xudong Lin, Ben Zhou, Manling Li, Haoyu Wang, Hongming Zhang, Xiaodong Yu, Alexander Dong, Zhenhailong Wang, Yi R. Fung, Piyush Mishra, Qing Lyu, Dídac Surís, Brian Chen, Susan W. Brown, Martha Palmer, Chris Callison-Burch, Carl Vondrick, Jiawei Han, Dan Roth, Shih-Fu Chang and Heng Ji
NAACL'21: System Demonstrations

Event-centric Natural Language Processing [PDF] [Slides]
Muhao Chen, Hongming Zhang, Qiang Ning, Manling Li, Heng Ji and Dan Roth
AAAI'21: Tutorial.

2020

Connecting the Dots: Event Graph Schema Induction with Path Language Modeling [PDF] [Code/Data] [Slides]
Manling Li, Qi Zeng, Ying Lin, Kyunghyun Cho, Heng Ji, Jonathan May, Nathanael Chambers and Clare Voss
EMNLP'20: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing.

GAIA: A Fine-grained Multimedia Knowledge Extraction System [PDF] [Code] [Video]
Manling Li*, Alireza Zareian*, Ying Lin, Xiaoman Pan, Spencer Whitehead, Brian Chen, Bo Wu, Heng Ji, Shih-Fu Chang, Clare R. Voss, Dan Napierski, Marjorie Freedman
ACL'20: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations. pp. 77–86
(Best Demo Paper Award at ACL2020)

Cross-media Structured Common Space for Multimedia Event Extraction [PDF] [Code] [Slides]
Manling Li*, Alireza Zareian*, Qi Zeng, Spencer Whitehead, Di Lu, Heng Ji, Shih-Fu Chang
ACL'20: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp.2557–2568

GAIA at SM-KBP 2020 - A Dockerized Multi-media Multi-lingual Knowledge Extraction, Clustering, Temporal Tracking and Hypothesis Generation System [PDF] [Project]
Manling Li, Ying Lin, Tuan Manh Lai, Xiaoman Pan, Haoyang Wen, Sha Li, etc %Zhenhailong Wang, Pengfei Yu, Lifu Huang, Di Lu, Qingyun Wang, Haoran Zhang, Qi Zeng, Chi Han, Zixuan Zhang, Yujia Qin, Xiaodan Hu, Nikolaus Parulian, Daniel Campos, Heng Ji, Brian Chen, Xudong Lin, Alireza Zareian, Amith Ananthram, Emily Allaway, Shih-Fu Chang, Kathleen McKeown, Yixiang Yao, Yifan Wang, Michael Spector, Mitchell DeHaven, Daniel Napierski, Marjorie Freedman, Pedro Szekely, Haidong Zhu, Ram Nevatia, Yang Bai, Yifan Wang, Ali Sadeghian, Haodi Ma, Daisy Zhe Wang
TAC-KBP: Text Analysis Conference Knowledge Base Population Workshop 2020 (Rank 1st in the leaderboard.)

2019

Keep Meeting Summaries on Topic: Abstractive Multi-Modal Meeting Summarization
[PDF] [Code]
Manling Li, Lingyu Zhang, Heng Ji, Rich Radke
ACL'19: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp.2190–2196

Multilingual Entity, Relation, Event and Human Value Extraction [PDF] [Code] [Video]
Manling Li, Ying Lin, Joe Hoover, Spencer Whitehead, Clare Voss, Morteza Dehghani, Heng Ji
NAACL'19: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (Demonstrations), pp.110–115

GAIA at SM-KBP 2019 - A Multi-media Multi-lingual KnowledgeExtraction and Hypothesis Generation System [PDF] [Project]
Manling Li, Ying Lin, Ananya Subburathinam, Spencer Whitehead, Xiaoman Pan, Di Lu, Qingyun Wang, Tongtao Zhang, Lifu Huang, Heng Ji. etc
TAC-KBP: Text Analysis Conference Knowledge Base Population Workshop 2019 (Rank 1st, with more than 10% higher than the second team.)

2018 and Before

Please see my full list at Google Scholar.

Tutorials

• Large Language Models for NLP in Finance
  Tutorial at ICAIF (ACM International Conference on AI in Finance) 2023. Nov, 2023.
  With Simerjot Kaur, Charese Smiley, Xiaomo Liu, Elena Kochkina, Mohammad Ghassemi, Reza Khanmohammadi.
  [Slides]
  

• Knowledge-Driven Vision-Language Encoding
  Tutorial at CVPR 2023. Jun, 2023.
  With Xudong Lin, Jie Lei, Carl Vondrick, Mohit Bansal, Heng Ji, and Shih-Fu Chang.
  [Website/Slides/Videos] [ReadingList]
  

• Knowledge-Driven Vision-Language Pretraining
  Tutorial at AAAI 2023. Feb, 2023.
  With Xudong Lin, Jie Lei, Mohit Bansal, Heng Ji, and Shih-Fu Chang.
  [Website/Slides/Videos] [ReadingList]
  

• New Frontiers of Information Extraction
  Tutorial at NAACL 2022. Jul, 2022.
  With Muhao Chen, Lifu Huang, Ben Zhou, Heng Ji, and Dan Roth.
  [Website/Slides/Videos] [PDF]

• Event-centric Natural Language Understanding
  Tutorial at ACL2021. Aug 2021.
  With Muhao Chen, Qiang Ning, Hongming Zhang, Heng Ji, Kathleen McKeown, and Dan Roth.
  [Website/Slides/Videos] [PDF]

• Event-centric Natural Language Processing
  Tutorial at AAAI2021. Feb, 2021.
  With Muhao Chen, Qiang Ning, Hongming Zhang, Heng Ji, and Dan Roth.
  [Website/Slides/Videos] [PDF]

Invited Talks

• Beyond the Beaten Path: Exploring the Role of Graphs in Multimodal Foundation Models
  Keynote Talk at NeurIPS 2023 Workshop on New Frontiers in Graph Learning. Dec 2023
  

• Towards Factulity in Information Access: Knowledge Foundation Models
  Nvidia Research. Oct 2023
  Renmin University. Oct 2023
  Fudan University. Oct 2023
  Zhejiang University. Oct 2023
  Tsinghua University. Oct 2023
  Peking University. Nov 2023
  

• LLMs for robotics: Modeling the Knowledge of the Physical World
  Stanford Vision and Learning Seminar. Oct 2023
  

• From Entity-Centric to Event-Centric Multimodal Event Knowledge Acquisition
  EE CS Rising Star Workshop, University of Texas at Austin, USA. Oct 2022.
  

• Towards Accurate Intelligent Analysis: Event-Centric Multimedia Knowledge Extraction
  DARPA Forward (Invite-Only), USA. Oct 2022.

• Event-Centric Multimedia Data Understanding
  Ohio State University, USA. Oct 2022.
  Singapore Management University, Singapore. Oct 2022.
  George Mason University, USA. Oct 2022.
  North Carolina State University, USA. Oct 2022.
  

• Multimedia Event Extraction: From Object-Centric to Event-Centric
  Virginia Tech, USA. Sept 2022.

• Event Knowledge Graph Construction
  LOGS Graph Reasoning Seminar. Aug 2022.

• Event Graph Structures in Vision-Language Understanding
  DataFun. Jun 2022.
  

• Connecting Vision and Text using Event Structures
  NewsBreak. Apr 2022.
  

• Memories as Repositories of Events: Structural Event Knowledge Acquisition
  University of Notre Dame. Feb 2022.
  

• Comprehensive Event Understanding in Multimedia Data
  USC ISI. Dec 2021.
  

• Structural Event Knowledge Acquisition from Multimedia Data
  UIUC NLP Seminar. Nov 2021.
  

• Event Extraction and Reasoning in Multimedia News Data
  Microsoft Research. Nov 2021.
  

• Improving Visual Event and Argument Role Understanding with Contrastive Image-Language Pretraining
  Microsoft Research. Aug 2021.
  

• Fine-Grained Knowledge Extraction System from Multimedia Data
  ai.science. Oct 2020.
  

• Event Understanding and Narration for Multimedia Data
  Intel MDI Research Lab. May 2020.
  

Organizing Committee:

• ACL 2025 (Virtual Infrastructure Chairs)
• EMNLP 2024 (Demonstration Chairs)
• Towords Knowledgeable Foundation Models @ ACL 2024 workshop
• Knowledge Discovery from Unstructured Data in Financial Services (KDF) Workshop

Area Chair:

• ACL (from 2023)
• EMNLP (from 2023)

Senior Program Committee:

• IJCAI, 2021

Program Committee:

• ARR (ACL Rolling Review, from 2021)
• ACL (from 2021)
• EMNLP (from 2021)
• NAACL-HLT (from 2021)
• AAAI (from 2021)
• WWW (from 2021)
• AKBC (2021)
• EACL (from 2021)
• KDD DI Workshop (2021)
• NLPCC (2021)
• COLING (from 2020)
• AACL (from 2020)
• CCL (from 2020)

Journal Reviewer:

• TACL
• TIST
• TKDE

Community Services:

• ACL Student Research Workshop (SRW) Mentor, ACL, 2024
• ACM Mentor, ACM Mentorship Program at UIUC, 2022
• CS Ambassador, UIUC CS Visit Day for Prospective Graduate Students, 2022
• Advising Assistant, UIUC PhD Orientation Seminar, 2021
• Graduate Student Representative, UIUC CS Visit Day, 2020

Awards

Microsoft Research Postdoc Fellowship 2023
EE CS Rising Star 2022
DARPA Riser 2022 (nominated by DARPA)
ACL 2020 Best Demo Paper Award
NAACL 2021 Best Demo Paper Award
Microsoft Research PhD Fellowship 2021
Mavis Future Faculty Fellow
C.L. and Jane Liu Award
National Scholarship

Academic and Scientific Competitions

Ranked 1st in NIST TAC Streaming Multimedia Knowledge Base Population (SM-KBP) 2020
Ranked 1st in NIST TAC Streaming Multimedia Knowledge Base Population (SM-KBP) 2019, with more than 10% absolute gains compared the second ranked team

Teaching Assistant:

• CS 598 Knowledge-driven Natural Language Generation, Spring 2022

Advising Assistant:

• CS 591 PhD Orientation Seminar, Fall 2021

Guest Lecturer:

• Event-Centric Multimedia Encoding
  CS 6604 Advanced Topics in Natural Language Processing
  Virginia Tech, Fall 2022
• Event-Centric Multimedia Encoding
  CSC 791&591: Advanced Topics in Efficient Deep Learning
  North Carolina State University, Fall 2022
• Knowledge-Driven Vision-Language Pretraining
  CS 546 Advanced Topics in Natural Language Processing
  UIUC, Fall 2022
• Recent Advances in Multimedia Encoding
  CS 546 Advanced Topics in Natural Language Processing
  UIUC, Fall 2022
• Timeline Summarization: Introducing Temporal Dimensions into Summarization
  CS 598 Knowledge Driven Natural Language Generation
  UIUC, Spring 2022
• Multimedia Encoding via Vision-Language Pretraining
  CS 546 Advanced Topics in Natural Language Processing
  UIUC, Fall 2021