Publications

Planning with the Views [Website][PDF][Code][Data][Models][tl;dr]
Kangrui Wang, Linjie Li, Zhengyuan Yang, Shiqi Chen, Zihan Wang, Li Fei-Fei, Jiajun Wu, Leonidas Guibas, Lijuan Wang, Manling Li

View PlanningSpatial ReasoningActive PerceptionEmbodied Agents

Learning from Trials and Errors: Reflective Test-Time Planning for Embodied LLMs [Website][PDF][Code][tl;dr]
Yining Hong, Huang Huang, Manling Li, Li Fei-Fei, Leonidas Guibas, Jiajun Wu, Yejin Choi
Best Paper Award at CVPR 2026 Workshop on Test-Time Scaling for Computer Vision (ViSCALE)

Test-Time PlanningSelf-ReflectionLearning from FailureEmbodied Planning

SkillCraft: Can LLM Agents Learn to Use Tools Skillfully? [Website][PDF][Code][tl;dr]
Shiqi Chen, Jingze Gai, Ruochen Zhou, Jinghan Zhang, Tongyao Zhu, Junlong Li, Kangrui Wang, Zihan Wang, Zhengyu Chen, Klara Kaleb, Ning Miao, Siyang Gao, Cong Lu, Manling Li, Junxian He, Yee Whye Teh

Tool UseSkill LearningLLM AgentsAgent Training

BAGEN: Are LLM Agents Budget-Aware? [Website][PDF][Code][Data][tl;dr]
Yuxiang Lin, Zihan Wang, Mengyang Liu, Yuxuan Shan, Longju Bai, Junyao Zhang, Xing Jin, Boshan Chen, Jinyan Su, Xingyao Wang, Jiaxin Pei, Manling Li

Budget-Aware AgentsLLM AgentsAgentic Reinforcement LearningTest-Time Compute

Theory of Space: Can Foundation Models Construct Spatial Beliefs Through Active Perception? [Website][PDF][Data][Code]
Pingyue Zhang*, Zihan Huang*, Yue Wang *, Jieyu Zhang*, Letian Xue, Zihan Wang, Qineng Wang, Keshigeyan Chandrasegaran, Yejin Choi, Ranjay Krishna, Ruohan Zhang, Jiajun Wu, Li Fei-Fei, Manling Li
ICLR 2026
Featured by MIT Tech Review China, Stanford AI Blog

Spatial IntelligenceActive PerceptionSpatial BeliefsWorld ModelsFoundation Models

ENACT: Evaluating Embodied Cognition with World Modeling of Egocentric Interaction [Website][PDF][Code][Data] [tl;dr]
Qineng Wang*, Wenlong Huang*, Yu Zhou, Hang Yin, Tianwei Bao, Jianwen Lyu, Weiyu Liu, Ruohan Zhang, Jiajun Wu, Li Fei-Fei, Manling Li
ICLR 2026
Outstanding Paper Award at ICLR 2026 Workshop on World Models
Outstanding Paper Award at ICLR 2026 Workshop on Lifelong Agents

Embodied CognitionWorld ModelingEgocentric InteractionEmbodied AI Benchmark

Spatial Mental Modeling from Limited Views [Website][PDF][Data][Code][tl;dr]
Qineng Wang*, Baiqiao Yin*, Pingyue Zhang, Jianshu Zhang, Kangrui Wang, Zihan Wang, Jieyu Zhang, Keshigeyan Chandrasegaran, Han Liu, Ranjay Krishna, Saining Xie, Jiajun Wu+, Li Fei-Fei+, Manling Li+
ICLR 2026
Best Paper Award at ICCV 2025 Workshop on Structural Priors for Vision
Best Paper Honorable Mention at NeurIPS 2025 Workshop on Language Agents and World Models (LAW)
The Best of ICCV 2025, featured by Voxel 51

Spatial ReasoningSpatial Mental ModelsCognitive MapsVision-Language ModelsSpatial Intelligence

ROSETTA: Constructing Code-Based Reward from Unconstrained Language Preference [Website][PDF][Data][Code][tl;dr]
Sanjana Srivastava*, Kangrui Wang*, Yung-Chieh Chan*, Tianyuan Dai, Manling Li, Ruohan Zhang, Mengdi Xu, Jiajun Wu, Li Fei-Fei
ICLR 2026
Best Paper Award at RSS 2025 on Continual Robot Learning from Humans

Reward ModelingCode-Based RewardsLanguage PreferencesAgentic Reinforcement Learning

RAGEN-2: Reasoning Collapse in Agentic Reinforcement Learning [Website][PDF][Code][Experimental Logs][tl;dr]
Zenus Wang*, Chi Gui*, Xing Jin*, Qineng Wang*, Licheng Liu*, Kangrui Wang, Shiqi Chen, Linjie Li, Zhengyuan Yang, Pingyue Zhang, Yiping Lu, Jiajun Wu, Li Fei-Fei, Lijuan Wang, Yejin Choi, Manling Li
ICML 2026 Oral
Best Poster Award at MMLS 2025 (Midwest Machine Learning Symposium)
2.7k+ Github Stars, Featured by MIT Tech Review, Lambda Partner Spotlight, VentureBeat, Medium, AI News, MarkTechPost, Business Leaders Review, etc.

Agentic Reinforcement LearningMulti-Turn Reinforcement LearningLLM AgentsReasoning CollapseRL Training Stability

ODESteer: A Unified ODE-Based Steering Framework for LLM Alignment [PDF][Website]
Hongjue Zhao*, Haosen Sun*, Jiangtao Kong, Xiaochang Li, Qineng Wang, Liwei Jiang, Qi Zhu, Tarek F. Abdelzaher, Yejin Choi, Manling Li+, Huajie Shao+ (equal advising)
ICLR 2026

LLM AlignmentActivation SteeringFlow MatchingControl TheoryAI Safety

Federated Agent Reinforcement Learning [Website][PDF]
Canyu Chen, Kangyu Zhu, Zhaorun Chen, Zhanhui Zhou, Shizhe Diao, Yiping Lu, Tian Li, Manling Li+, Dawn Song+
Best Paper Award at AAAI 2026 Workshop on Trustworthy Agentic Systems
Oustanding Paper Award at AAAI 2026 Workshop on Personalization in the Era of Large Foundation Models

Federated LearningAgentic Reinforcement LearningLLM AgentsPrivacy-Preserving Training

ActionEQA: Action Interface for Embodied Question Answering [PDF]
Tianwei Bao, Qineng Wang, Kangrui Wang, Mingkai Deng, Guangyi Liu, Jiayuan Mao, Lawrence Birnbaum, Zhiting Hu, Eric P. Xing, Zhaoran Wang, Manling Li
TMLR 2026

Embodied Question AnsweringAction InterfaceEmbodied AgentsSpatial Reasoning

Fairness Failure Modes of Multimodal LLMs [PDF]
Canyu Chen*, Anglin Cai*, Joan Nwatu, Yale Li, Han Liu, Jessica Hullman, Rada Mihalcea, Kathleen McKeown, Manling Li
Preprint, 2026

FairnessMultimodal LLMsBias EvaluationTrustworthy AI

Weak-to-Strong Generalization with Failure Trajectories [PDF]
Ruimeng Ye, Zihan Wang, Yang Xiao, Zinan Ling, Manling Li, Bo Hui
ICLR 2026

Weak-to-Strong GeneralizationScalable OversightAlignmentFailure Trajectories

Your Language Model Secretly Contains Personality Subnetworks [PDF][Code]
Ruimeng Ye, Zihan Wang, Zinan Ling, Yang Xiao, Manling Li, Xiaolong Ma, Bo Hui
ICLR 2026

InterpretabilityPersonality SubnetworksModel EditingLLM Alignment

VAGEN: Reinforcing World Model Reasoning for Multi-Turn VLM Agents [Website][PDF][Blog][Code][tl;dr]
Kangrui Wang*, Pingyue Zhang*, Zihan Wang*, Yaning Gao*, Linjie Li*, Qineng Wang, Chi Wan, Hanyang Chen, Yiping Lu, Zhengyuan Yang, Lijuan Wang, Ranjay Krishna, Jiajun Wu, Li Fei-Fei, Yejin Choi, Manling Li
NeurIPS 2025
Featured by MIT Tech Review, Stanford AI Blog

World Model ReasoningWorld Model Reinforcement LearningMulti-Turn Reinforcement LearningRL for VisionVLM AgentsAgentic Reinforcement Learning

WorldAgen: Unified State-Action Prediction with Test-Time World Model Training [PDF]
Chi Wan*, Kangrui Wang*, Yuan Si, Pingyue Zhang, Manling Li
AAAI 2026

World ModelsTest-Time TrainingState-Action PredictionEmbodied Agents

Exploring Diffusion Transformer Designs via Grafting [Website][PDF][Blog][Code][tl;dr]
Keshigeyan Chandrasegaran*, Michael Poli*, Daniel Y. Fu, Dongjun Kim, Lea M. Hadzic, Manling Li, Agrim Gupta, Stefano Massaroli, Azalia Mirhoseini, Juan Carlos Niebles, Stefano Ermon, Li Fei-Fei
NeurIPS 2025 (Oral, Top 0.36%)

Diffusion TransformersModel GraftingArchitecture DesignGenerative Models

EmbodiedBench: Comprehensive Benchmarking Multi-modal Large Language Models for Vision-Driven Embodied Agents [Website][PDF][Code][tl;dr]
Rui Yang, Hanyang Chen, Junyu Zhang, Mark Zhao, Cheng Qian, Kangrui Wang, Qineng Wang, Teja Venkat Koripella, Marziyeh Movahedi, Manling Li, Heng Ji, Huan Zhang, Tong Zhang
ICML 2025 (Oral, Top 1%)

Embodied AgentsMultimodal LLMsEmbodied AI BenchmarkVision-Driven Agents

Why Is Spatial Reasoning Hard for VLMs? An Attention Mechanism Perspective on Focus Areas [PDF][Code][Data]
Shiqi Chen, Tongyao Zhu, Ruochen Zhou, Jinghan Zhang, Siyang Gao, Juan Carlos Niebles, Mor Geva, Junxian He, Jiajun Wu, Manling Li
ICML 2025

Spatial ReasoningAttention MechanismsVLM InterpretabilityAdaptive Decoding

Bring Reason to Vision: Understanding Perception and Reasoning through Model Merging [Website][PDF][Code][Data]
Shiqi Chen, Jinghan Zhang, Tongyao Zhu, Wei Liu, Siyang Gao, Miao Xiong, Manling Li, Junxian He
ICML 2025

Model MergingPerception and ReasoningVision-Language Models

SyncMind: Measuring Agent Out-of-Sync Recovery in Collaborative Software Engineering [Website][PDF][Code][Data]
Xuehang Guo, Xingyao Wang, Yangyi Chen, Sha Li, Chi Han, Manling Li, Heng Ji
ICML 2025

Software Engineering AgentsAgent CollaborationLLM AgentsBenchmark

Internalizing World Models via Self-Play Finetuning for Agentic RL [PDF][Code]
Shiqi Chen, Tongyao Zhu, Zian Wang, Jinghan Zhang, Kangrui Wang, Siyang Gao, Teng Xiao, Yee Whye Teh, Junxian He, Manling Li
2025

World ModelsSelf-Play Fine-TuningAgentic Reinforcement LearningLLM Agents

ERA: Embodied Reasoning Agents via Reinforcement Learning [Website][PDF][Code][Data]
Hanyang Chen, Mark Zhao, Rui Yang, Qinwei Ma, Ke Yang, Jiarui Yao, Kangrui Wang, Hao Bai, Zhenhailong Wang, Rui Pan, Mengchao Zhang, Jose Barreiros, Aykut Onol, ChengXiang Zhai, Heng Ji, Manling Li, Huan Zhang, Tong Zhang
2025

Embodied ReasoningReinforcement LearningRL for VisionEmbodied AgentsVLM Agents

SENTINEL: A Multi-Level Formal Framework for Safety Evaluation of LLM-based Embodied Agents [PDF]
Simon Sinong Zhan, Yao Liu, Philip Wang, Zinan Wang, Qineng Wang, Zhian Ruan, Xiangyu Shi, Xinyu Cao, Frank Yang, Kangrui Wang, Huajie Shao, Manling Li, Qi Zhu
2025

Safety EvaluationFormal MethodsEmbodied Agent SafetyTemporal Logic

T*: Re-thinking Temporal Search for Long-Form Video Understanding [Website][PDF][Data][Code]
Jinhui Ye*, Zihan Wang*, Haosen Sun, Keshigeyan Chandrasegaran, Zane Durante, Cristobal Eyzaguirre, Yonatan Bisk, Juan Carlos Niebles, Ehsan Adeli, Li Fei-Fei, Jiajun Wu, Manling Li
CVPR 2025, Oral at ICCV 2025 Workshop on Long Multi-Scene Video Foundations

Long-Form Video UnderstandingTemporal SearchVideo-Language ModelsKeyframe Selection

LayoutVLM: Differentiable Optimization of 3D Layout via Vision-Language Models [Website][PDF][Code]
Fan-Yun Sun, Weiyu Liu, Siyi Gu, Dylan Lim, Goutam Bhat, Federico Tombari, Manling Li, Nick Haber, Jiajun Wu
CVPR 2025

3D Scene LayoutSpatial ReasoningDifferentiable OptimizationVision-Language Models

Chain-of-Action: Faithful and Multimodal Question Answering through Large Language Models [PDF]
Zhenyu Pan, Haozheng Luo, Manling Li, Han Liu
ICLR 2025

Multimodal Question AnsweringFaithful ReasoningLLM AgentsRetrieval-Augmented Generation

Visually Descriptive Language Modeling for Vector Graphics Reasoning [PDF][Website][Code]
Zhenhailong Wang, Joy Hsu, Xingyao Wang, Kuan-Hao Huang, Manling Li, Jiajun Wu, Heng Ji
TMLR

Vector Graphics ReasoningVisual ReasoningSymbolic Representations

The Law of Knowledge Overshadowing: Towards Understanding, Predicting and Preventing LLM Hallucination [PDF]
Yuji Zhang, Sha Li, Cheng Qian, Jiateng Liu, Pengfei Yu, Chi Han, Yi Fung, Kathleen McKeown, ChengXiang Zhai, Manling Li, Heng Ji
ACL 2025 Findings

HallucinationKnowledge OvershadowingLLM TruthfulnessHallucination Prediction

ACLED-DS: A Large Multilingual Expert-Annotated Abstractive Event Dataset for the Real World [PDF]
Sina Semnani, Pingyue Zhang, Wanyue Zhai, Haozhuo Li, Ryan Beauchamp, Trey Billing, Katayoun Kishi, Manling Li, Monica Lam
ACL 2025 Findings

Event ExtractionMultilingual DatasetConflict AnalysisInformation Extraction

Chain-of-Experts: Unlocking the Communication Power of MoEs [PDF][Blog][Code][tl;dr]
Zihan Wang, Rui Pan, Jiarui Yao, Róbert Csordás, Linjie Li, Lu Yin, Jiajun Wu, Tong Zhang, Manling Li, Shiwei Liu

Mixture-of-ExpertsSparse ArchitecturesExpert CommunicationLLM Efficiency

Foundation Models Meet Embodied Agents [Website/Slides/Videos]
Manling Li, Yunzhu Li, Jiayuan Mao, Wenlong Huang
AAAI 2025: Tutorial
NAACL 2025: Tutorial
ICCV 2025: Tutorial

TutorialFoundation ModelsEmbodied AgentsReasoning and Planning

Towards Safe Multi-Modal Learning: Unique Challenges and Future Directions [Website/Slides/Videos]
Xi Li, Manling Li, Muchao Ye
ICCV 2025: Tutorial
CVPR 2026: Tutorial

TutorialMultimodal SafetyTrustworthy AI

The Lifecycle of Knowledge in Large Language Models: Memorization, Editing, and Beyond [Website/Slides/Videos]
Manling Li, Sha Li, Yuji Zhang, Chi Han, Heng Ji
AAAI 2025: Tutorial

TutorialKnowledge EditingLLM MemorizationKnowledge Lifecycle

Embodied Agent Interface: Benchmarking LLMs for Embodied Decision Making [Website][PDF][Code][Data][Docker][PyPi][Doc]
Manling Li^*, Shiyu Zhao^*, Qineng Wang^*, Kangrui Wang^*, Yu Zhou^*, Sanjana Srivastava, Cem Gokmen, Tony Lee, Li Erran Li, Ruohan Zhang, Weiyu Liu, Percy Liang, Li Fei-Fei, Jiayuan Mao, Jiajun Wu
NeurIPS 2024 Benchmark Track (Oral, Top 0.6%)
Best Paper Award at SoCal NLP 2024, Top 0.4%

Embodied Decision MakingLLM AgentsEmbodied AI BenchmarkReasoning and Planning

HourVideo: 1-Hour Video-Language Understanding [Website][PDF][Data][Code]
Keshigeyan Chandrasegaran, Agrim Gupta, Taran Kota, Lea M. Hadzic, Jimming He, Cristobal Eyzaguirre, Zane Durante, Manling Li, Jiajun Wu, Li Fei-Fei
NeurIPS 2024 Benchmark Track

Long-Form Video UnderstandingVideo-Language BenchmarkEgocentric Video

IKEA Manuals at Work: 4D Grounding of Assembly Instructions on Internet Videos [Website][PDF][Data][Code]
Yunong Liu, Weiyu Liu, Shubh Khanna, Cristobal Eyzaguirre, Manling Li, Juan Carlos Niebles, Vineeth Ravi, Saumitra Mishra, Jiajun Wu
NeurIPS 2024 Benchmark Track

4D GroundingAssembly InstructionsProcedural Video Understanding

LM-Steer: Word Embeddings Are Steers for Language Models [Website][PDF][Code][Live Demo][Slides][Poster]
Chi Han, Jialiang Xu, Manling Li, Yi Fung, Chenkai Sun, Nan Jiang, Tarek Abdelzaher, Heng Ji
ACL 2024
(Outstanding Paper Award at ACL 2024)

LLM SteeringControllable GenerationWord EmbeddingsAlignmentInterpretability

Why Does New Knowledge Create Messy Ripple Effects in LLMs? [PDF]
Jiaxin Qin, Zixuan Zhang, Chi Han, Pengfei Yu, Manling Li, Heng Ji
EMNLP 2024

Knowledge EditingRipple EffectsLLM Knowledge

Deep Concept Injection for Zero-shot Multimodal Reasoning [PDF]
Xudong Lin, Manling Li, Richard Zemel, Heng Ji, Shih-Fu Chang
EMNLP 2024

Zero-Shot Multimodal ReasoningConcept InjectionVideo Question Answering

MentalArena: Self-play Training of Language Models for Diagnosis and Treatment of Mental Health Disorders [PDF]
Cheng Li, May Fung, Qingyun Wang, Chi Han, Manling Li, Jindong Wang, Heng Ji
2024

Self-Play TrainingMental Health AILLM Applications

Can LLMs Produce Faithful Explanations For Fact-checking? Towards Faithful Explainable Fact-Checking via Multi-Agent Debate [PDF]
Kyungha Kim*, Sangyun Lee*, Kung-Hsiang Huang*, Hou Pong Chan, Manling Li, Heng Ji
2024

Fact-CheckingFaithful ExplanationsMulti-Agent Debate

InfoPattern: Unveiling Information Propagation Patterns in Social Media [PDF]
Chi Han*, Jialiang Xu*, Manling Li* , Hanning Zhang*, Tarek Abdelzaher, Heng Ji
2023

Information PropagationSocial Media AnalysisComputational Social Science

SmartBook: AI-Assisted Situation Report Generation [PDF]
Revanth Gangi Reddy, Yi Fung, Qi Zeng, Manling Li, Zihan Wang, Paul Sullivan, Heng Ji
2023

Situation Report GenerationAI-Assisted WritingEvent Understanding

Controlling Object Existence Hallucinations in Large Vision Language Models [PDF]
Bohan Zhai, Shijia Yang, Chenfeng Xu, Sheng Shen, Kurt Keutzer, Chunyuan Li, Manling Li
2023

Object HallucinationHallucination MitigationVision-Language Models

Event-centric Multimodal Knowledge Acquisition [PDF]
Manling Li
Thesis Committee: Heng Ji, Jiawei Han, Chengxiang Zhai, Shih-Fu Chang, Kyunghyun Cho
Thesis

Multimodal Knowledge AcquisitionEvent-Centric NLPInformation Extraction

ViStruct: Visual Structural Knowledge Extraction via Curriculum Guided Code-Vision Representation [PDF]
Yangyi Chen, Xingyao Wang, Manling Li, Derek Hoiem, Heng Ji
EMNLP 2023

Visual Knowledge ExtractionCurriculum LearningCode-Vision Representation

Defining a New NLP Playground [PDF]
Sha Li, Chi Han, Pengfei Yu, Carl Edwards, Manling Li, Xingyao Wang, Yi Fung, Charles Yu, Joel R. Tetreault, Eduard H Hovy, Heng Ji
EMNLP 2023 Findings

NLP Research AgendaLarge Language ModelsPosition Paper

Knowledge-Driven Vision-Language Encoding [Website/Slides/Videos]
Manling Li, Xudong Lin, Jie Lei, Mohit Bansal, Carl Vondrick, Shih-Fu Chang, Heng Ji
CVPR 2023: Tutorial

TutorialVision-Language PretrainingKnowledge-Driven Learning

Towards Fast Adaptation of Pretrained Contrastive Models for Multi-channel Video-Language Retrieval [PDF] [Code]
Xudong Lin, Simran Tiwari, Shiyuan Huang, Manling Li, Mike Zheng Shou, Heng Ji, Shih-Fu Chang
CVPR 2023

Video-Language RetrievalContrastive LearningEfficient Adaptation

Open Visual Knowledge Extraction via Relation-Oriented Multimodality Model Prompting [PDF][Website]
Hejie Cui, Xinyu Fang, Zihan Zhang, Ran Xu, Xuan Kan, Xin Liu, Manling Li, Yangqiu Song, Carl Yang
NeurIPS 2023

Visual Knowledge ExtractionMultimodal PromptingRelation Extraction

Non-Sequential Graph Script Induction via Multimedia Grounding [PDF]
Yu Zhou^†, Sha Li, Manling Li, Xudong Lin, Shih-Fu Chang, Mohit Bansal and Heng Ji
ACL 2023 (^† denotes supervised undergraduate)

Graph Script InductionProcedural KnowledgeMultimedia Grounding

A Language First Approach to Procedure Planning [PDF]
Jiateng Liu^†, Sha Li, Zhenhailong Wang, Manling Li, Heng Ji
ACL 2023 Findings (^† denotes supervised undergraduate)

Procedure PlanningTask PlanningLanguage Priors

Open-Domain Hierarchical Event Schema Induction by Incremental Prompting and Verification [PDF]
Sha Li, Ruining Zhao^†, Manling Li, Heng Ji, Chris Callison-Burch and Jiawei Han
ACL 2023 (^† denotes supervised undergraduate)

Event Schema InductionHierarchical SchemasLLM Prompting

Multimedia Generative Script Learning for Task Planning [PDF]
Qingyun Wang, Manling Li, Hou Pong Chan, Lifu Huang, Julia Hockenmaier, Girish Chowdhary and Heng Ji
ACL 2023 Findings

Script LearningTask PlanningProcedural Knowledge

Learning to Decompose Visual Features with Latent Textual Prompts [PDF] [Code]
Feng Wang^†, Manling Li, Xudong Lin, Hairong Lv, Alexander Schwing, Heng Ji
ICLR 2023 (^† denotes supervised undergraduate)

Visual Feature DecompositionPrompt LearningCLIP

Knowledge-Driven Vision-Language Pretraining [Website/Slides/Videos]
Manling Li, Xudong Lin, Jie Lei, Mohit Bansal, Shih-Fu Chang, Heng Ji
AAAI 2023: Tutorial

TutorialVision-Language PretrainingKnowledge Integration

Video Event Extraction via Tracking Visual States of Arguments [PDF] [Code]
Guang Yang^†, Manling Li, Jiajie Zhang, Xudong Lin, Shih-Fu Chang, Heng Ji
AAAI 2023 (^† denotes supervised undergraduate)

Video Event ExtractionVisual State TrackingMultimodal Information Extraction

ADEPT: A DEbiasing PrompT Framework [PDF] [Code]
Ke Yang^†, Charles Yu, Yi Fung, Manling Li, Heng Ji
AAAI 2023 (^† denotes supervised undergraduate)

DebiasingPrompt TuningFairness in NLP

Language Models with Image Descriptors are Strong Few-Shot Video-Language Learners [PDF] [Code]
Zhenhailong Wang^†*,Manling Li*, Ruochen Xu, Luowei Zhou, Jie Lei, Xudong Lin, Shuohang Wang, Ziyi Yang, Chenguang Zhu, Derek Hoiem, Shih-Fu Chang, Mohit Bansal, Heng Ji
NeurIPS'22 (equal contribution)

Few-Shot Video-Language LearningImage DescriptorsLLM Prompting

CLIP-Event:Connecting Vision and Text with Event Structures [PDF] [Data] [Code]
Manling Li, Ruochen Xu, Shuohang Wang, Xudong Lin, Chenguang Zhu, Xuedong Huang, Heng Ji, Shih-Fu Chang
CVPR'22 (Oral, Top 4.1%)

Event StructuresVision-Language PretrainingZero-Shot TransferContrastive Learning

COVID-19 Claim Radar: A Structured Claim Extraction and Tracking System [PDF] [Code] [Demo] [Video]
Manling Li, Revanth Gangi Reddy, Ziqi Wang, Yi-Shyuan Chiang, Tuan M. Lai, Pengfei Yu, Zixuan Zhang,Heng Ji
ACL'22 Demo

Claim ExtractionMisinformation TrackingNLP System

Event Schema Induction with Double Graph Autoencoders [PDF] [Code]
Xiaomeng Jin^†, Manling Li and Heng Ji
NAACL'22

Event Schema InductionGraph AutoencodersEvent Graphs

New Frontiers of Information Extraction [Website/Slides/Videos]
Muhao Chen, Lifu Huang, Manling Li, Ben Zhou, Heng Ji
NAACL'22: Tutorial

TutorialInformation Extraction

MuMuQA: Multimedia Multi-Hop News Question Answering via Cross-Media Knowledge Extraction and Grounding [PDF] [Data]
Revanth Gangi Reddy^†, Xilin Rui^†, Manling Li, Xudong Lin, Haoyang Wen, Jaemin Cho, Lifu Huang, Mohit Bansal, Avi Sil, Shih-Fu Chang, Alexander Schwing, Heng Ji
AAAI'22

Multimedia Question AnsweringMulti-Hop ReasoningCross-Media Grounding

The Future is not One-dimensional: Complex Event Schema Induction by Graph Modeling for Event Prediction [PDF] [Data]
Manling Li, Sha Li, Zhenhailong Wang, Lifu Huang, Kyunghyun Cho, Heng Ji, Jiawei Han and Clare Voss
EMNLP'21

Complex Event SchemasTemporal GraphsEvent Prediction

Timeline Summarization based on Event Graph Compression via Time-Aware Optimal Transport [PDF] [Data]
Manling Li, Tengfei Ma, Mo Yu, Lingfei Wu, Tian Gao, Heng Ji and Kathleen McKeown
EMNLP'21

Timeline SummarizationEvent GraphsOptimal Transport

Joint Multimedia Event Extraction from Video and Article [PDF] [Data]
Brian Chen, Xudong Lin, Christopher Thomas, Manling Li, Shoya Yoshida, Lovish Chum, Heng Ji and Shih-Fu Chang
EMNLP'21 Findings

Multimedia Event ExtractionVideo-Text UnderstandingInformation Extraction

Event-centric Natural Language Processing [Website/Slides/Videos]
Muhao Chen, Hongming Zhang, Qiang Ning, Manling Li, Heng Ji, Kathleen McKeown and Dan Roth
ACL'21, AAAI'21: Tutorial.

TutorialEvent-Centric NLP

COVID-19 Literature Knowledge Graph Construction and Drug Repurposing Report Generation [PDF] [Code/Data]
Qingyun Wang, Manling Li, Xuan Wang, Nikolaus Parulian, Guangxing Han, Jiawei Ma, Jingxuan Tu, Ying Lin, Haoran Zhang, Weili Liu, Aabhas Chauhan, Yingjun Guan, Bangzheng Li, Ruisong Li, Xiangchen Song, Heng Ji, Jiawei Han, Shih-Fu Chang, James Pustejovsky, David Liem, Ahmed Elsayed, Martha Palmer, Jasmine Rah, Clare Voss, Cynthia Schneider, Boyan Onyshkevych
NAACL'21: System Demonstrations
(Best Demo Paper Award at NAACL2021)

Knowledge Graph ConstructionDrug RepurposingScientific NLP

RESIN: A Dockerlized Schema-Guided Cross-document Cross-lingual Cross-media Information Extraction and Event Tracking System [PDF] [Code]
Haoyang Wen, Ying Lin, Tuan M. Lai, Xiaoman Pan, Sha Li, Xudong Lin, Ben Zhou, Manling Li, Haoyu Wang, Hongming Zhang, Xiaodong Yu, Alexander Dong, Zhenhailong Wang, Yi R. Fung, Piyush Mishra, Qing Lyu, Dídac Surís, Brian Chen, Susan W. Brown, Martha Palmer, Chris Callison-Burch, Carl Vondrick, Jiawei Han, Dan Roth, Shih-Fu Chang and Heng Ji
NAACL'21: System Demonstrations

Schema-Guided ExtractionCross-LingualCross-MediaInformation Extraction System

Connecting the Dots: Event Graph Schema Induction with Path Language Modeling [PDF] [Code/Data]
Manling Li, Qi Zeng, Ying Lin, Kyunghyun Cho, Heng Ji, Jonathan May, Nathanael Chambers and Clare Voss
EMNLP'20: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing.

Event Schema InductionPath Language ModelingEvent Graphs

GAIA: A Fine-grained Multimedia Knowledge Extraction System [PDF] [Code] [Video]
Manling Li*, Alireza Zareian*, Ying Lin, Xiaoman Pan, Spencer Whitehead, Brian Chen, Bo Wu, Heng Ji, Shih-Fu Chang, Clare R. Voss, Dan Napierski, Marjorie Freedman
ACL'20: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations. pp. 77–86
(Best Demo Paper Award at ACL2020)

Multimedia Knowledge ExtractionKnowledge GraphsFine-Grained Entities

Cross-media Structured Common Space for Multimedia Event Extraction [PDF] [Code] [Slides]
Manling Li*, Alireza Zareian*, Qi Zeng, Spencer Whitehead, Di Lu, Heng Ji, Shih-Fu Chang
ACL'20: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp.2557–2568

Multimedia Event ExtractionCross-Media Common SpaceWeakly Supervised Learning

GAIA at SM-KBP 2020 - A Dockerized Multi-media Multi-lingual Knowledge Extraction, Clustering, Temporal Tracking and Hypothesis Generation System [PDF] [Project]
Manling Li, Ying Lin, Tuan Manh Lai, Xiaoman Pan, Haoyang Wen, Sha Li, etc %Zhenhailong Wang, Pengfei Yu, Lifu Huang, Di Lu, Qingyun Wang, Haoran Zhang, Qi Zeng, Chi Han, Zixuan Zhang, Yujia Qin, Xiaodan Hu, Nikolaus Parulian, Daniel Campos, Heng Ji, Brian Chen, Xudong Lin, Alireza Zareian, Amith Ananthram, Emily Allaway, Shih-Fu Chang, Kathleen McKeown, Yixiang Yao, Yifan Wang, Michael Spector, Mitchell DeHaven, Daniel Napierski, Marjorie Freedman, Pedro Szekely, Haidong Zhu, Ram Nevatia, Yang Bai, Yifan Wang, Ali Sadeghian, Haodi Ma, Daisy Zhe Wang
TAC-KBP: Text Analysis Conference Knowledge Base Population Workshop 2020 (Rank 1st in the leaderboard.)

Multimedia Knowledge ExtractionMultilingual Information ExtractionKnowledge Base Population

Keep Meeting Summaries on Topic: Abstractive Multi-Modal Meeting Summarization
[PDF]
Manling Li, Lingyu Zhang, Heng Ji, Rich Radke
ACL'19: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp.2190–2196

Meeting SummarizationMultimodal SummarizationAbstractive Summarization

Multilingual Entity, Relation, Event and Human Value Extraction [PDF] [Code] [Video]
Manling Li, Ying Lin, Joe Hoover, Spencer Whitehead, Clare Voss, Morteza Dehghani, Heng Ji
NAACL'19: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (Demonstrations), pp.110–115

Multilingual Information ExtractionEvent ExtractionHuman Values

GAIA at SM-KBP 2019 - A Multi-media Multi-lingual KnowledgeExtraction and Hypothesis Generation System [PDF] [Project]
Manling Li, Ying Lin, Ananya Subburathinam, Spencer Whitehead, Xiaoman Pan, Di Lu, Qingyun Wang, Tongtao Zhang, Lifu Huang, Heng Ji, Alireza Zareian, Hassan Akbari, Brian Chen, Bo Wu, Emily Allaway, Shih-Fu Chang, Kathleen McKeown, Yixiang Yao, Jennifer Chen, Eric Berquist, Kexuan Sun, Xujun Peng, Ryan Gabbard Marjorie Freedman, Pedro Szekely, T.K. Satish Kumar, Arka Sadhu, Ram Nevatia, Miguel Rodriguez, Yifan Wang, Yang Bai, Ali Sadeghian, Daisy Zhe Wang
TAC-KBP: Text Analysis Conference Knowledge Base Population Workshop 2019 (Rank 1st, with more than 10% higher than the second team.)

Multimedia Knowledge ExtractionMultilingual Information ExtractionKnowledge Base Population