Hi, there! I’m Manling. I am an assistant professor at the Computer Science department of Northwestern University (NU) in Chicago/Evanston, affiliated with Cognitive Science and Center for Robotics and Biosystems, where I direct the Machine Learning and Language (MLL) Lab. I am also an Amazon Scholar working on conversational agents. Prior to this, I was a postdoc at Stanford University and fortunate to be advised by Prof Jiajun Wu and mentored by Prof Fei-Fei Li with support partially from Microsoft Research Postdoc Fellowship. I obtained my Ph.D. degree in Computer Science at University of Illinois Urbana-Champaign in 2023, where I was fortunate to have Prof Heng Ji as my advisor and grateful for the mentorship from Prof Shih-Fu Chang, Prof Kyunghyun Cho and Prof Jiawei Han, and for receiving the support from Microsoft Research PhD Fellowship. I am a recipient of MIT Tech Review 35 Under 35, ACL Inaugural Best Desseratation Award Hornorable Mention in 2025, DARPA Riser in 2022, and a EE CS Rising Star in 2022, Microsoft Research PhD Fellowship in 2021, etc. Our work on multimodal reasoning was awarded ACL'24 Outstanding Paper Award, ACL'20 Best Demo Paper Award, NAACL'21 Best Demo Paper Award, etc. I served as organizing committee of ACL'25 (virtual co-chairs), NAACL'25 (publication co-chairs), EMNLP'24 (demo co-chairs), etc, and organized the Workshops/Challenges on Foundation Models Meet Embodied Agents at NeurIPS'25, CVPR'25, etc, the Workshops on Knowledgeable Foundation Models at ACL'25, ACL'24, AAAI'25, etc.

I aim to make AI and LLMs beneficial for humans, working at the intersection of language, vision, robotics, and their impact on society. I focus on Reasoning, Planning, Compositionality, with applications in Embodied AI and AI for science. At the core of my research, I aim to make machines to interact with physical world via multimodal data (Language + X, where X can be robotics, vision, audio, etc). The ultimate goal of my research is to promote trustworthiness and truthfulness in through a structured view that is easily explainable, highly compositional, and capable of long-horizon reasoning.

Prospective students: I have several PhD positions in Fall 2026 and intern positions. We have amazing new faculty at NU working on robotics and foundation models, please check out Ruohan! He also has several PhD positions in Fall 2026.

>> If you are interested in PhD at Northwestern, please apply to NorthwesternCS. Due to the large amount of emails, I aplogize that I will not able to reply to individual emails (Please note that non-reply does NOT indicate non-interesting, largely means emails got missed or I unfortunately did not get time to check such emails before applications). Please choose me as a potential advisor in the application, and I will check every application carefully in late Dec and do interviews in Jan-Mar.
>> If you are interested in doing research internship with our group, please feel free to talk to any of the PhD students to join their projects, and many of them are looking for collaborators. The best way is to submit this form and drop an email to limanling.ai@gmail.com. This mailbox has been checked more frequently.
>> Prospective_Students_English
>> Prospective_Students_Chinese
>> Proud of what our students have achieved in the very first year!

Junior PhD/master/undergraduate students: I will dedicate 30 minutes each week to offer guidance/suggestions/mentorship, especially for students from underrepresented groups or whoever is in need. If you would like to chat about life, career path, graduate school applications, or research ideas related to AI/ML, feel free to file the form to schedule a meeting.

Interests

Publications

Talks

Tutorials

Invited Talks

  • Columbia University. Oct 2025

  • University of Pennsylvania. Oct 2025

  • ICCV Workshop on Multimodal Reasoning for Agentic Intelligence. Oct 2025

  • ICCV Workshop on Multimodal Spatial Intelligence. Oct 2025

  • ICCV Workshop on Structural Priors for Vision. Oct 2025

  • ICCV Workshop on LongVid-Foundations. Oct 2025

  • RAGEN: Training Agents by Reinforcing Reasoning
    Agent AI Summit at Berkeley. Aug 2025.
    University of Edinburgh CHAI Seminar Series. Sept 2025
    Agentic AI Frontier Seminar, Sept 2025

  • See, Think, Act: Agent Training By Reinforcement Reasoning
    Cross Future AI Summit, Jul 2025

  • Why is Spatial Concept Learning Hard?
    CVPR 2025 Workshop on Visual Concepts, Jul 2025.

  • Training Agents with World Model Reasoning
    Apple Workshop on Reasoning and Planning, Jul 2025

  • LLMs for Embodied Decision Making
    ACC 2025 Workshop on LLMs in Control Design and Decision Making

  • RAGEN: Training Agents by Reinforcing Reasoning
    Google Deepmind. May 2025
    UIUC NLP Seminar. Apr 2025

  • Reasoning and Planning with Physical World
    Guest Lecture at UMich EECS 692 Advanced Artificial Intelligence. Apr 2025

  • Agent Training Under a MDP Formulation
    AAAI 2025 New Faculty Highlights. Feb 2025
    AAAI 2025 Workshop on LM4Plan. Feb 2025
    AAAI 2025 Bridge on Foundation Models and Planning. Feb 2025

  • Embodied Agent Interface: LLMs and VLMs for Embodied Reasoning and Planning
    SFU @ NeurIPS 2024. Dec 2024

  • LLMs for Embodied Agents
    EMNLP 2024 Birds of Feather. Nov 2024

  • Customizing Large Language Models to Embodied Agent interacting with Embodied Envioronments
    EMNLP 2024 CustomNLP4U Workshop. Nov 2024

  • From Large Language Models to Large Agent Models
    Keynote at Amazon-Illinois Center on AI for Interactive Conversational Experiences Fall Research Symposium 2024. Sept 2024

  • Reasoning and Planning with Physical World Knowledge
    2024 Allerton Conference on Communication, Control, and Computing. Sept 2024

  • Embodied Agent Interface: LLMs for Embodied Decision Making
    TTIC Multimodal AI Workshop 2024. Aug 2024

  • Multimodal Knowledge for Social Good
    Summer Institute in Computational Social Science 2024. Aug 2024

  • Reasoning, Planning and Compositionality in Multimodality
    SpLU-RoboNLP 2024 Workshop at ACL. Jul 2024

  • Visually Descriptive Language Modeling for Document Intelligence
    Adobe Research. Jul 2024

  • From Large Language Models to Large Agent Models
    Apple NLU Workshop 2024. Jun 2024

  • From Words to Worlds: A Close Look to Diffusion Models (through an NLP Lens)
    UIUC NLP Seminar. Jun 2024

  • The Missing Knowledge in LLMs to Interact with the Physical World
    Midwest Machine Learning Symposium 2024. May 2024

  • Beyond the Beaten Path: Exploring the Role of Graphs in Multimodal Foundation Models
    Keynote Talk at NeurIPS 2023 Workshop on New Frontiers in Graph Learning. Dec 2023

  • LLMs for robotics: Modeling the Knowledge of the Physical World
    Stanford Vision and Learning Seminar. Oct 2023

  • Knowledge Foundation Models
    Adobe Research. Oct 2023

  • Modeling the Semantics of the Physical World
    Stanford CogAI. Jun 2023

  • Towards Factuality in Information Access: Multimodal Knowledge Acqusition and Reasoning [Slides]
    Carnegie Mellon University, LTI. Feb 2023
    Northwestern University, CS. Feb 2023
    Northeastern University, ECE. Feb 2023
    Purdue University, CS. Feb 2023
    Rice University, CS. Feb 2023
    Virginia Tech, CS. Feb 2023
    Max Planck Institute. Feb 2023
    UVA, CS. Mar 2023
    MBZUAI. Mar 2023
    U Washington St Louis, CS. Mar 2023
    University of Toronto, CS+ECE. Mar 2023
    UC San Diego, ECE. Feb 2023
    UC Davis, CS. Mar 2023
    UC Los Angeles, ECE. Apr 2023

  • From Entity-Centric to Event-Centric Multimodal Event Knowledge Acquisition
    EE CS Rising Star, University of Texas at Austin, USA. Oct 2022.

  • Towards Accurate Intelligent Analysis: Event-Centric Multimedia Knowledge Extraction
    DARPA Forward (Invite-Only), USA. Oct 2022.

  • Event-Centric Multimedia Data Understanding
    Ohio State University, USA. Oct 2022.
    Singapore Management University, Singapore. Oct 2022.
    George Mason University, USA. Oct 2022.
    North Carolina State University, USA. Oct 2022.

  • Multimedia Event Extraction: From Object-Centric to Event-Centric
    Virginia Tech, USA. Sept 2022.

  • Event Knowledge Graph Construction
    LOGS Graph Reasoning Seminar. Aug 2022.

  • Event Graph Structures in Vision-Language Understanding
    DataFun. Jun 2022.

  • Connecting Vision and Text using Event Structures
    NewsBreak. Apr 2022.

  • Memories as Repositories of Events: Structural Event Knowledge Acquisition
    University of Notre Dame. Feb 2022.

  • Comprehensive Event Understanding in Multimedia Data
    USC ISI. Dec 2021.

  • Structural Event Knowledge Acquisition from Multimedia Data
    UIUC NLP Seminar. Nov 2021.

  • Event Extraction and Reasoning in Multimedia News Data
    Microsoft Research. Nov 2021.

  • Improving Visual Event and Argument Role Understanding with Contrastive Image-Language Pretraining
    Microsoft Research. Aug 2021.

  • Fine-Grained Knowledge Extraction System from Multimedia Data
    ai.science. Oct 2020.

  • Event Understanding and Narration for Multimedia Data
    Intel MDI Research Lab. May 2020.

Academic Service

Organizing Committee:

  • ACL 2025 (Virtual Infrastructure Chairs)
  • NAACL 2025 (Publication Chairs) summary
  • EMNLP 2024 (Demonstration Chairs) summary
  • Foundation Models Meet Embodied Agents @ CVPR 2025 workshop
  • Towords Knowledgeable Foundation Models @ ACL 2025 workshop
  • Towords Knowledgeable Foundation Models @ AAAI 2025 workshop
  • Towords Knowledgeable Foundation Models @ ACL 2024 workshop
  • Knowledge Discovery from Unstructured Data in Financial Services (KDF) @ SIGIR 2023 Workshop

Senior Area Chair:

  • EMNLP (from 2025)

Area Chair:

  • ACL (from 2023)
  • EMNLP (from 2023)
  • NAACL (from 2024)
  • COLM (from 2025)

Program Committee:

  • ARR (ACL Rolling Review, from 2021)
  • ACL (from 2021)
  • EMNLP (from 2021)
  • NAACL-HLT (from 2021)
  • AAAI (from 2021)
  • WWW (from 2021)
  • AKBC (2021)
  • EACL (from 2021)
  • KDD DI Workshop (2021)
  • NLPCC (2021)
  • COLING (from 2020)
  • AACL (from 2020)
  • CCL (from 2020)

Journal Reviewer:

  • TACL
  • TIST
  • TKDE

Community Services:

  • ACL Student Research Workshop (SRW) Mentor, ACL, 2024
  • ACM Mentor, ACM Mentorship Program at UIUC, 2022
  • CS Ambassador, UIUC CS Visit Day for Prospective Graduate Students, 2022
  • Advising Assistant, UIUC PhD Orientation Seminar, 2021
  • Graduate Student Representative, UIUC CS Visit Day, 2020

Awards

Awards

ACL Inaugural Best Desseratation Award Honorable Mention
MIT Tech Review 35 Under 35 Global List
Best Poster Award at MMLS 2025
Best Paper Award at RSS 2025 workshop on Continual Robot Learning from Humans
Outstanding Paper Award at ACL 2024
Best Paper Award at SoCal NLP 2024
Best Demo Paper Award at NAACL 2021
Best Demo Paper Award at ACL 2020
AAAI 2025 New Faculty Highlights
Microsoft Research Postdoc Fellowship 2023
EE CS Rising Star 2022
DARPA Riser 2022
Microsoft Research PhD Fellowship 2021
Mavis Future Faculty Fellow
C.L. and Jane Liu Award
National Scholarship

Academic and Scientific Competitions

Ranked 1st in NIST TAC Streaming Multimedia Knowledge Base Population (SM-KBP) 2020
Ranked 1st in NIST TAC Streaming Multimedia Knowledge Base Population (SM-KBP) 2019, with more than 10% absolute gains compared the second ranked team

Teaching

Instructor:

Guest Lecturer:

  • Navigating Up and Down in the Job Searching
    COMP_SCI 496 Academic Job Search
    NU, Fall 2023
  • Event-Centric Multimedia Encoding
    CS 6604 Advanced Topics in Natural Language Processing
    Virginia Tech, Fall 2022
  • Knowledge-Driven Vision-Language Pretraining
    CS 546 Advanced Topics in Natural Language Processing
    UIUC, Fall 2022
  • Recent Advances in Multimedia Encoding
    CS 546 Advanced Topics in Natural Language Processing
    UIUC, Fall 2022
  • Timeline Summarization: Introducing Temporal Dimensions into Summarization
    CS 598 Knowledge Driven Natural Language Generation
    UIUC, Spring 2022
  • Multimedia Encoding via Vision-Language Pretraining
    CS 546 Advanced Topics in Natural Language Processing
    UIUC, Fall 2021

Students

It is a great pleasure to work with such talented young people. I am grateful for the trust that they have placed in me.
Proud of what our students have achieved in the very first year!

PhDs

Avatar

Zihan Wang

PhD (2024-)

RAGEN

Avatar

Pingyue Zhang

PhD (2024-)

Spatial Reasoning

Avatar

Kangrui Wang

PhD (2024-)

Reasoning and Planning

Avatar

Qineng Wang

PhD (2024-)

Foundation Models for Embodied Agents

Avatar

Jianshu Zhang

PhD (2025-)

Vision-Language Models, co-advised w/ Prof Han Liu

Avatar

Canyu Chen

PhD (2025-)

Truthful, Safe and Responsible LLMs

Avatar

Shiqi Chen

Visiting PhD (2024-)

Avatar

Lei (Max) Zhang

Visiting PhD (2025-)

Masters

Avatar

Haosen Sun

Master @ NU (2024-)

Avatar

Anabella Isaro

Master @ NU (2025-)

Avatar

Xuehang Guo

Master @ UIUC (2024-)

Avatar

Tianwei Bao

Master @ NU (2025-)

Avatar

Chi Wan

Master @ NU (2024-)

Undergraduates

Avatar

Gulsimo Osimi

BA-MS @ NU (2025-)

Avatar

Ishan Mukherjee

Undergraduate @ NU (2025-)

Avatar

Kefan Yu

Undergraduate @ NU (2025-)

Avatar

Baiqiao Yin

Undergraduate (2024-)

Past Members

Shiyu Zhao, 2024, Master @ Stanford
Gasol Sun, 2024, Master @ Xiamen → PhD @ UIUC
Yihe Wang, 2024, Undergraduate @ Tsinghua, now exchange at UCLA
Jinhui Ye, 2024, Master @ HKUST-GZ → PhD @ HKUST
Zhengyu Hu, 2024, Master @ HKUST-GZ
Yu (Bryan) Zhou, 2022/2024, Undergraduate @ UCLA → PhD @ UCLA
Hanning Zhang, 2023, Undergraduate @ HKUST → Master @ UIUC
Guang Yang, 2022, Undergraduate @ Tsinghua → PhD @ UW
Ke Yang, 2022, Undergraduate @ Tsinghua → PhD @ UIUC
Feng Wang, 2022, Master @ Tsinghua → PhD @ JHU
Jialiang Xu, 2022, Undergraduate @ UIUC → Master @ Stanford
Ruining Zhao, 2022, Undergraduate @ UIUC, published her first paper at ACL 2023
Jiateng Liu, 2022, Undergraduate @ ZJU → Master @ UIUC
Xilin Rui, 2021, Undergraduate @ Tsinghua, published her first paper at AAAI 2022
Zhiying Zhu, 2021, Undergraduate @ ZJU → Master @ CMU
Zhenhailong Wang, 2020-2022, Master @ UIUC → PhD @ UIUC