Hi, there! I’m Manling. I am an incoming assistant professor at the Computer Science department of Northwestern University and a postdoc at Stanford University working with Prof Jiajun Wu. I obtained my Ph.D. degree in Computer Science at University of Illinois Urbana-Champaign in 2023, working with Prof. Heng Ji as a member of the Blender NLP lab. At the core of my research in Knowledge Foundation Models, I aim to equip machines with factual knowledge extraction and reasoning from multimodal data (Language + X, where X can be images, videos, robotics, audio, etc). The ultimate goal of my research is to promote factuality and truthfulness in information access, through a structured knowledge view that is easily explainable, highly compositional, and capable of long-horizon reasoning.

Bio: Her work on multimodal knowledge extraction won the ACL'20 Best Demo Paper Award, and the work on scientific information extraction from COVID literature won NAACL'21 Best Demo Paper Award. She was a recipient of Microsoft Research PhD Fellowship in 2021. She was selected as a DARPA Riser in 2022 (nominated by DARPA), and a EE CS Rising Star in 2022. She was awarded C.L. Dave and Jane W.S. Liu Award, and has been selected as a Mavis Future Faculty Fellow. She led 19 students to develop the UIUC information extraction system and ranked 1st in DARPA AIDA evaluation each phase. She has more than 30 publications on multimodal knowledge extraction and reasoning, and gave tutorials about event-centric multimodal knowledge at ACL'21, AAAI'21, NAACL'22, AAAI'23, etc.

Prospective students: I have 2-5 PhD positions and 1 postdoc position in Fall 2024 or later. Please apply to NorthwesternCS, which is open until Dec 15. Please also submit this form. Due to the large amount of emails, I may miss some of them. The best way to reach me is by filling out the form.

Junior PhD/master/undergraduate students: I will dedicate 30 minutes each week to offer guidance/suggestions/mentorship, especially for students from underrepresented groups or whoever is in need. If you would like to chat about life, career path, graduate school applications, or research ideas related to AI/ML, feel free to file the form to schedule a meeting.


  • [2024-02] Serve as EMNLP 2024 Organizing Committee (Demonstration Chairs).
  • [2024-02] Got my very first PhD student Zihan! Super excited! Thanks for the trust and I will do my best to be the best advisor like Heng.
  • [2023-12] Keynote Talk “Beyond the Beaten Path: Exploring the Role of Graphs in Multimodal Foundation Models” at NeurIPS 2023 Workshop on New Frontiers in Graph Learning. Panel discussion with Jiawei Han, Marinka Zitnik, Yanqi Zhou, Mohammad Shoeybi.
  • [2023-10] Will organize “Towards Knowledgeable Language Models @ ACL24 Workshop” Call for Papers, submission deadline in May 20, 2024.
  • [2023-08] Joined Stanford Vision and Learning lab as a postdoc working with Prof Jiajun Wu. Excited!
  • [2023-07] Served as a session chair for multimodality at ACL 2023.
  • [2023-06] Will serve as an Area Chair at EMNLP 2023.
  • [2023-05] Will join Northwestern Computer Science as a tenure-track assistant professor.
  • [2022-12] Will serve as an Area Chair at ACL 2023.
  • [2022-10] Will present at EE CS Rising Star workshop in UT Austin on Oct 25-26 as a Riser.
  • [2022-10] Will present at DARPA Forward event in OSU on Oct 4-5 as a DARPA Riser.
  • [2022-07] Will give a tutorial at NAACL T3: New Frontiers of Information Extraction at 9:00–12:30 on July 10th (Location: Columbia D) with Muhao Chen, Lifu Huang, Ben Zhou, Heng Ji and Dan Roth. Welcome to our tutorial!

Full Publications

Please find my latest publications on Google Scholar.

Our workshop is calling for papers! Towards Knowledgeable Language Models @ ACL 2024 Workshop [Website]


Advances in Information Access
Sha Li, Manling Li, Heng Ji
Ongoing (World Scientifc Publishing)

Natural Language Processing: A Modern Perspective
Bang Liu, Yu Chen, Manling Li, Heng Ji, Lingfei Wu
Ongoing (Cambridge University Press)

Papers (* denotes equal contribution, denotes mentored students)


Text-based Reasoning About Vector Graphics [PDF][Website][Code]
Zhenhailong Wang, Joy Hsu, Xingyao Wang, Kuan-Hao Huang, Manling Li, Jiajun Wu, Heng Ji

Can LLMs Produce Faithful Explanations For Fact-checking? Towards Faithful Explainable Fact-Checking via Multi-Agent Debate [PDF]
Kyungha Kim*, Sangyun Lee*, Kung-Hsiang Huang *, Hou Pong Chan, Manling Li, Heng Ji


Event-centric Multimodal Knowledge Acquisition [PDF]
Manling Li
Thesis Committee: Heng Ji, Jiawei Han, Chengxiang Zhai, Shih-Fu Chang, Kyunghyun Cho

LM-Switch: Flexible Language Model Conditioning [PDF]
Chi Han, Jialiang Xu, Manling Li, Yi Fung, Chenkai Sun, Nan Jiang, Tarek Abdelzaher, Heng Ji

Controlling Object Existence Hallucinations in Large Vision Language Models [PDF]
Bohan Zhai, Shijia Yang, Chenfeng Xu, Sheng Shen, Kurt Keutzer, Manling Li

Deep Concept Injection for Zero-shot Multimodal Reasoning [PDF]
Xudong Lin, Manling Li, Richard Zemel, Heng Ji, Shih-Fu Chang

ViStruct: Visual Structural Knowledge Extraction via Curriculum Guided Code-Vision Representation [PDF]
Yangyi Chen, Xingyao Wang, Manling Li, Derek Hoiem, Heng Ji
EMNLP 2023

Defining a New NLP Playground [PDF]
Sha Li, Chi Han, Pengfei Yu, Carl Edwards, Manling Li, Xingyao Wang, Yi Fung, Charles Yu, Joel R. Tetreault, Eduard H Hovy, Heng Ji
EMNLP 2023 Findings

Knowledge-Driven Vision-Language Encoding [Website]
Manling Li, Xudong Lin, Jie Lei, Mohit Bansal, Carl Vondrick, Shih-Fu Chang, Heng Ji
CVPR 2023: Tutorial

Towards Fast Adaptation of Pretrained Contrastive Models for Multi-channel Video-Language Retrieval [PDF] [Code]
Xudong Lin, Simran Tiwari, Shiyuan Huang, Manling Li, Mike Zheng Shou, Heng Ji, Shih-Fu Chang
CVPR 2023

Open Visual Knowledge Extraction via Relation-Oriented Multimodality Model Prompting [Website]
Hejie Cui, Xinyu Fang, Zihan Zhang, Ran Xu, Xuan Kan, Xin Liu, Manling Li, Yangqiu Song, Carl Yang
NeurIPS 2023

Non-Sequential Graph Script Induction via Multimedia Grounding [PDF]
Yu Zhou, Sha Li, Manling Li, Xudong Lin, Shih-Fu Chang, Mohit Bansal and Heng Ji
ACL 2023 ( denotes supervised undergraduate)

A Language First Approach to Procedure Planning [PDF]
Jiateng Liu, Sha Li, Zhenhailong Wang, Manling Li, Heng Ji
ACL 2023 Findings ( denotes supervised undergraduate)

Open-Domain Hierarchical Event Schema Induction by Incremental Prompting and Verification [PDF]
Sha Li, Ruining Zhao, Manling Li, Heng Ji, Chris Callison-Burch and Jiawei Han
ACL 2023 ( denotes supervised undergraduate)

Multimedia Generative Script Learning for Task Planning [PDF]
Qingyun Wang, Manling Li, Hou Pong Chan, Lifu Huang, Julia Hockenmaier, Girish Chowdhary and Heng Ji
ACL 2023 Findings

Learning to Decompose Visual Features with Latent Textual Prompts [PDF] [Code]
Feng Wang, Manling Li, Xudong Lin, Hairong Lv, Alexander Schwing, Heng Ji
ICLR 2023 ( denotes supervised undergraduate)

Knowledge-Driven Vision-Language Pretraining [PDF] [Website]
Manling Li, Xudong Lin, Jie Lei, Mohit Bansal, Shih-Fu Chang, Heng Ji
AAAI 2023: Tutorial

Video Event Extraction via Tracking Visual States of Arguments [PDF] [Code]
Guang Yang, Manling Li, Jiajie Zhang, Xudong Lin, Shih-Fu Chang, Heng Ji
AAAI 2023 ( denotes supervised undergraduate)

ADEPT: A DEbiasing PrompT Framework [PDF] [Code]
Ke Yang, Charles Yu, Yi Fung, Manling Li, Heng Ji
AAAI 2023 ( denotes supervised undergraduate)


Language Models with Image Descriptors are Strong Few-Shot Video-Language Learners [PDF] [Code]
Zhenhailong Wang*,Manling Li*, Ruochen Xu, Luowei Zhou, Jie Lei, Xudong Lin, Shuohang Wang, Ziyi Yang, Chenguang Zhu, Derek Hoiem, Shih-Fu Chang, Mohit Bansal, Heng Ji
NeurIPS'22 (equal contribution)

CLIP-Event:Connecting Vision and Text with Event Structures [PDF] [Data] [Code]
Manling Li, Ruochen Xu, Shuohang Wang, Xudong Lin, Chenguang Zhu, Xuedong Huang, Heng Ji, Shih-Fu Chang
CVPR'22 (Oral)

COVID-19 Claim Radar: A Structured Claim Extraction and Tracking System [PDF] [Code] [Demo] [Video]
Manling Li, Revanth Gangi Reddy, Ziqi Wang, Yi-Shyuan Chiang, Tuan M. Lai, Pengfei Yu, Zixuan Zhang,Heng Ji
ACL'22 Demo

Event Schema Induction with Double Graph Autoencoders [PDF] [Code]
Xiaomeng Jin, Manling Li and Heng Ji

New Frontiers of Information Extraction [PDF] [Website] [Slides] [Videos]
Muhao Chen, Lifu Huang, Manling Li, Ben Zhou, Heng Ji
NAACL'22: Tutorial

MuMuQA: Multimedia Multi-Hop News Question Answering via Cross-Media Knowledge Extraction and Grounding [PDF] [Data]
Revanth Gangi Reddy, Xilin Rui, Manling Li, Xudong Lin, Haoyang Wen, Jaemin Cho, Lifu Huang, Mohit Bansal, Avi Sil, Shih-Fu Chang, Alexander Schwing, Heng Ji


The Future is not One-dimensional: Complex Event Schema Induction by Graph Modeling for Event Prediction [PDF] [Data]
Manling Li, Sha Li, Zhenhailong Wang, Lifu Huang, Kyunghyun Cho, Heng Ji, Jiawei Han and Clare Voss

Timeline Summarization based on Event Graph Compression via Time-Aware Optimal Transport [PDF] [Data]
Manling Li, Tengfei Ma, Mo Yu, Lingfei Wu, Tian Gao, Heng Ji and Kathleen McKeown

Joint Multimedia Event Extraction from Video and Article [PDF] [Data]
Brian Chen, Xudong Lin, Christopher Thomas, Manling Li, Shoya Yoshida, Lovish Chum, Heng Ji and Shih-Fu Chang
EMNLP'21 Findings

Event-centric Natural Language Processing [PDF] [Slides]
Muhao Chen, Hongming Zhang, Qiang Ning, Manling Li, Heng Ji, Kathleen McKeown and Dan Roth
ACL'21: Tutorial.

COVID-19 Literature Knowledge Graph Construction and Drug Repurposing Report Generation [PDF] [Code/Data]
Qingyun Wang, Manling Li, Xuan Wang, Nikolaus Parulian, Guangxing Han, Jiawei Ma, Jingxuan Tu, Ying Lin, Haoran Zhang, Weili Liu, Aabhas Chauhan, Yingjun Guan, Bangzheng Li, Ruisong Li, Xiangchen Song, Heng Ji, Jiawei Han, Shih-Fu Chang, James Pustejovsky, David Liem, Ahmed Elsayed, Martha Palmer, Jasmine Rah, Clare Voss, Cynthia Schneider, Boyan Onyshkevych
NAACL'21: System Demonstrations
(Best Demo Paper Award at NAACL2021)

RESIN: A Dockerlized Schema-Guided Cross-document Cross-lingual Cross-media Information Extraction and Event Tracking System [PDF] [Code]
Haoyang Wen, Ying Lin, Tuan M. Lai, Xiaoman Pan, Sha Li, Xudong Lin, Ben Zhou, Manling Li, Haoyu Wang, Hongming Zhang, Xiaodong Yu, Alexander Dong, Zhenhailong Wang, Yi R. Fung, Piyush Mishra, Qing Lyu, Dídac Surís, Brian Chen, Susan W. Brown, Martha Palmer, Chris Callison-Burch, Carl Vondrick, Jiawei Han, Dan Roth, Shih-Fu Chang and Heng Ji
NAACL'21: System Demonstrations

Event-centric Natural Language Processing [PDF] [Slides]
Muhao Chen, Hongming Zhang, Qiang Ning, Manling Li, Heng Ji and Dan Roth
AAAI'21: Tutorial.


Connecting the Dots: Event Graph Schema Induction with Path Language Modeling [PDF] [Code/Data] [Slides]
Manling Li, Qi Zeng, Ying Lin, Kyunghyun Cho, Heng Ji, Jonathan May, Nathanael Chambers and Clare Voss
EMNLP'20: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing.

GAIA: A Fine-grained Multimedia Knowledge Extraction System [PDF] [Code] [Video]
Manling Li*, Alireza Zareian*, Ying Lin, Xiaoman Pan, Spencer Whitehead, Brian Chen, Bo Wu, Heng Ji, Shih-Fu Chang, Clare R. Voss, Dan Napierski, Marjorie Freedman
ACL'20: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations. pp. 77–86
(Best Demo Paper Award at ACL2020)

Cross-media Structured Common Space for Multimedia Event Extraction [PDF] [Code] [Slides]
Manling Li*, Alireza Zareian*, Qi Zeng, Spencer Whitehead, Di Lu, Heng Ji, Shih-Fu Chang
ACL'20: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp.2557–2568

GAIA at SM-KBP 2020 - A Dockerized Multi-media Multi-lingual Knowledge Extraction, Clustering, Temporal Tracking and Hypothesis Generation System [PDF] [Project]
Manling Li, Ying Lin, Tuan Manh Lai, Xiaoman Pan, Haoyang Wen, Sha Li, etc %Zhenhailong Wang, Pengfei Yu, Lifu Huang, Di Lu, Qingyun Wang, Haoran Zhang, Qi Zeng, Chi Han, Zixuan Zhang, Yujia Qin, Xiaodan Hu, Nikolaus Parulian, Daniel Campos, Heng Ji, Brian Chen, Xudong Lin, Alireza Zareian, Amith Ananthram, Emily Allaway, Shih-Fu Chang, Kathleen McKeown, Yixiang Yao, Yifan Wang, Michael Spector, Mitchell DeHaven, Daniel Napierski, Marjorie Freedman, Pedro Szekely, Haidong Zhu, Ram Nevatia, Yang Bai, Yifan Wang, Ali Sadeghian, Haodi Ma, Daisy Zhe Wang
TAC-KBP: Text Analysis Conference Knowledge Base Population Workshop 2020 (Rank 1st in the leaderboard.)

UIUC TAC2020 RUFES System Description [PDF] [Project]
Revanth Gangi Reddy, Manling Li, Haoyang Wen and Heng Ji
TAC-KBP: Text Analysis Conference Knowledge Base Population Workshop 2020


Keep Meeting Summaries on Topic: Abstractive Multi-Modal Meeting Summarization
[PDF] [Code]
Manling Li, Lingyu Zhang, Heng Ji, Rich Radke
ACL'19: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp.2190–2196

Multilingual Entity, Relation, Event and Human Value Extraction [PDF] [Code] [Video]
Manling Li, Ying Lin, Joe Hoover, Spencer Whitehead, Clare Voss, Morteza Dehghani, Heng Ji
NAACL'19: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (Demonstrations), pp.110–115

The Unobtrusive Group Interaction (UGI) Corpus [PDF] [Code]
Indrani Bhattacharya, Michael Foley, Ni Zhang, Tongtao Zhang, Christine Ku, Cameron Mine, Manling Li, Heng Ji, etc.
ACM MMSys'19: ACM Multimedia Systems Conference 2019, pp.249-254

GAIA at SM-KBP 2019 - A Multi-media Multi-lingual KnowledgeExtraction and Hypothesis Generation System [PDF] [Project]
Manling Li, Ying Lin, Ananya Subburathinam, Spencer Whitehead, Xiaoman Pan, Di Lu, Qingyun Wang, Tongtao Zhang, Lifu Huang, Heng Ji. etc
TAC-KBP: Text Analysis Conference Knowledge Base Population Workshop 2019 (Rank 1st, with more than 10% higher than the second team.)

2018 and Before

Link Prediction in Knowledge Graphs: A Hierarchy-Constrained Approach [PDF] [Code]
Manling Li, Yuanzhuo Wang, Denghui Zhang, Yantao Jia, Xueqi Cheng
IEEE Transactions on Big Data, pp.1-14
Special Issue on “Knowledge Graphs: Techniques and Applications”


Hierarchy-Based Link Prediction in Knowledge Graphs [PDF] [Poster]
Manling Li, Yantao Jia, Yuanzhuo Wang, Jingyuan Li, Xueqi Cheng.
WWW 2016: Proceedings of the 25th International Conference Companion on World Wide Web, pp.77-78 (abstract paper)

Predicting Links and Their Building Time: A Path-Based Method [PDF] [Poster]
Manling Li, Yantao Jia, Yuanzhuo Wang, Zeya Zhao, Xueqi Cheng.
AAAI 2016: Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence, pp.4228-4229 (abstract paper)



Invited Talks

  • Beyond the Beaten Path: Exploring the Role of Graphs in Multimodal Foundation Models
    Keynote Talk at NeurIPS 2023 Workshop on New Frontiers in Graph Learning. Dec 2023

  • Towards Factulity in Information Access: Knowledge Foundation Models
    Nvidia Research. Oct 2023
    Renmin University. Oct 2023
    Fudan University. Oct 2023
    Zhejiang University. Oct 2023
    Tsinghua University. Oct 2023
    Peking University. Nov 2023

  • LLMs for robotics: Modeling the Knowledge of the Physical World
    Stanford Vision and Learning Seminar. Oct 2023

  • From Entity-Centric to Event-Centric Multimodal Event Knowledge Acquisition
    EE CS Rising Star Workshop, University of Texas at Austin, USA. Oct 2022.

  • Towards Accurate Intelligent Analysis: Event-Centric Multimedia Knowledge Extraction
    DARPA Forward (Invite-Only), USA. Oct 2022.

  • Event-Centric Multimedia Data Understanding
    Ohio State University, USA. Oct 2022.
    Singapore Management University, Singapore. Oct 2022.
    George Mason University, USA. Oct 2022.
    North Carolina State University, USA. Oct 2022.

  • Multimedia Event Extraction: From Object-Centric to Event-Centric
    Virginia Tech, USA. Sept 2022.

  • Event Knowledge Graph Construction
    LOGS Graph Reasoning Seminar. Aug 2022.

  • Event Graph Structures in Vision-Language Understanding
    DataFun. Jun 2022.

  • Connecting Vision and Text using Event Structures
    NewsBreak. Apr 2022.

  • Memories as Repositories of Events: Structural Event Knowledge Acquisition
    University of Notre Dame. Feb 2022.

  • Comprehensive Event Understanding in Multimedia Data
    USC ISI. Dec 2021.

  • Structural Event Knowledge Acquisition from Multimedia Data
    UIUC NLP Seminar. Nov 2021.

  • Event Extraction and Reasoning in Multimedia News Data
    Microsoft Research. Nov 2021.

  • Improving Visual Event and Argument Role Understanding with Contrastive Image-Language Pretraining
    Microsoft Research. Aug 2021.

  • Fine-Grained Knowledge Extraction System from Multimedia Data
    ai.science. Oct 2020.

  • Event Understanding and Narration for Multimedia Data
    Intel MDI Research Lab. May 2020.

Academic Service

Organizing Committee:

  • EMNLP 2024 (Demonstration Chairs)
  • Towords Knowledgeable Foundation Models @ ACL 2024 workshop
  • Knowledge Discovery from Unstructured Data in Financial Services (KDF) Workshop

Area Chair:

  • ACL, 2023
  • EMNLP, 2023
  • ACL, 2024

Senior Program Committee:

  • IJCAI, 2021

Program Committee:

  • ARR (ACL Rolling Review, from 2021)
  • ACL (from 2021)
  • EMNLP (from 2021)
  • NAACL-HLT (from 2021)
  • AAAI (from 2021)
  • WWW (from 2021)
  • AKBC (2021)
  • EACL (from 2021)
  • KDD DI Workshop (2021)
  • NLPCC (2021)
  • COLING (from 2020)
  • AACL (from 2020)
  • CCL (from 2020)

Journal Reviewer:

  • IEEE MultiMedia
  • Journal of Artificial Intelligence Research
  • Journal of Advances in Information Technology

Community Services:

  • ACM Mentor, ACM Mentorship Program at UIUC, 2022
  • CS Ambassador, UIUC CS Visit Day for Prospective Graduate Students, 2022
  • Advising Assistant, UIUC PhD Orientation Seminar, 2021
  • Graduate Student Representative, UIUC CS Visit Day, 2020
  • Co-Chair, “Source” Department Technology Association in University of Science and Technology Beijing, 2014



Microsoft Research Postdoc Fellowship 2023
EE CS Rising Star 2022
DARPA Riser 2022 (nominated by DARPA)
ACL 2020 Best Demo Paper Award
NAACL 2021 Best Demo Paper Award
Microsoft Research PhD Fellowship 2021
Mavis Future Faculty Fellow
C.L. and Jane Liu Award
National Scholarship

Academic and Scientific Competitions

Ranked 1st in NIST TAC Streaming Multimedia Knowledge Base Population (SM-KBP) 2020
Ranked 1st in NIST TAC Streaming Multimedia Knowledge Base Population (SM-KBP) 2019, with more than 10% absolute gains compared the second ranked team


Teaching Assistant:

  • CS 598 Knowledge-driven Natural Language Generation, Spring 2022

Advising Assistant:

  • CS 591 PhD Orientation Seminar, Fall 2021

Guest Lecturer:

  • Event-Centric Multimedia Encoding
    CS 6604 Advanced Topics in Natural Language Processing
    Virginia Tech, Fall 2022
  • Event-Centric Multimedia Encoding
    CSC 791&591: Advanced Topics in Efficient Deep Learning
    North Carolina State University, Fall 2022
  • Knowledge-Driven Vision-Language Pretraining
    CS 546 Advanced Topics in Natural Language Processing
    UIUC, Fall 2022
  • Recent Advances in Multimedia Encoding
    CS 546 Advanced Topics in Natural Language Processing
    UIUC, Fall 2022
  • Timeline Summarization: Introducing Temporal Dimensions into Summarization
    CS 598 Knowledge Driven Natural Language Generation
    UIUC, Spring 2022
  • Multimedia Encoding via Vision-Language Pretraining
    CS 546 Advanced Topics in Natural Language Processing
    UIUC, Fall 2021