Embodied Spatial Reasoning

Planning with the Views [Website][PDF][Code][Data][Models][tl;dr]
Kangrui Wang, Linjie Li, Zhengyuan Yang, Shiqi Chen, Zihan Wang, Li Fei-Fei, Jiajun Wu, Leonidas Guibas, Lijuan Wang, Manling Li

View PlanningSpatial ReasoningActive PerceptionEmbodied Agents

Learning from Trials and Errors: Reflective Test-Time Planning for Embodied LLMs [Website][PDF][Code][tl;dr]
Yining Hong, Huang Huang, Manling Li, Li Fei-Fei, Leonidas Guibas, Jiajun Wu, Yejin Choi
Best Paper Award at CVPR 2026 Workshop on Test-Time Scaling for Computer Vision (ViSCALE)

Test-Time PlanningSelf-ReflectionLearning from FailureEmbodied Planning

Theory of Space: Can Foundation Models Construct Spatial Beliefs Through Active Perception? [Website][PDF][Data][Code]
Pingyue Zhang*, Zihan Huang*, Yue Wang *, Jieyu Zhang*, Letian Xue, Zihan Wang, Qineng Wang, Keshigeyan Chandrasegaran, Yejin Choi, Ranjay Krishna, Ruohan Zhang, Jiajun Wu, Li Fei-Fei, Manling Li
ICLR 2026
Featured by MIT Tech Review China, Stanford AI Blog

Spatial IntelligenceActive PerceptionSpatial BeliefsWorld ModelsFoundation Models

ENACT: Evaluating Embodied Cognition with World Modeling of Egocentric Interaction [Website][PDF][Code][Data] [tl;dr]
Qineng Wang*, Wenlong Huang*, Yu Zhou, Hang Yin, Tianwei Bao, Jianwen Lyu, Weiyu Liu, Ruohan Zhang, Jiajun Wu, Li Fei-Fei, Manling Li
ICLR 2026
Outstanding Paper Award at ICLR 2026 Workshop on World Models
Outstanding Paper Award at ICLR 2026 Workshop on Lifelong Agents

Embodied CognitionWorld ModelingEgocentric InteractionEmbodied AI Benchmark

Spatial Mental Modeling from Limited Views [Website][PDF][Data][Code][tl;dr]
Qineng Wang*, Baiqiao Yin*, Pingyue Zhang, Jianshu Zhang, Kangrui Wang, Zihan Wang, Jieyu Zhang, Keshigeyan Chandrasegaran, Han Liu, Ranjay Krishna, Saining Xie, Jiajun Wu+, Li Fei-Fei+, Manling Li+
ICLR 2026
Best Paper Award at ICCV 2025 Workshop on Structural Priors for Vision
Best Paper Honorable Mention at NeurIPS 2025 Workshop on Language Agents and World Models (LAW)
The Best of ICCV 2025, featured by Voxel 51

Spatial ReasoningSpatial Mental ModelsCognitive MapsVision-Language ModelsSpatial Intelligence

ActionEQA: Action Interface for Embodied Question Answering [PDF]
Tianwei Bao, Qineng Wang, Kangrui Wang, Mingkai Deng, Guangyi Liu, Jiayuan Mao, Lawrence Birnbaum, Zhiting Hu, Eric P. Xing, Zhaoran Wang, Manling Li
TMLR 2026

Embodied Question AnsweringAction InterfaceEmbodied AgentsSpatial Reasoning

Exploring Diffusion Transformer Designs via Grafting [Website][PDF][Blog][Code][tl;dr]
Keshigeyan Chandrasegaran*, Michael Poli*, Daniel Y. Fu, Dongjun Kim, Lea M. Hadzic, Manling Li, Agrim Gupta, Stefano Massaroli, Azalia Mirhoseini, Juan Carlos Niebles, Stefano Ermon, Li Fei-Fei
NeurIPS 2025 (Oral, Top 0.36%)

Diffusion TransformersModel GraftingArchitecture DesignGenerative Models

EmbodiedBench: Comprehensive Benchmarking Multi-modal Large Language Models for Vision-Driven Embodied Agents [Website][PDF][Code][tl;dr]
Rui Yang, Hanyang Chen, Junyu Zhang, Mark Zhao, Cheng Qian, Kangrui Wang, Qineng Wang, Teja Venkat Koripella, Marziyeh Movahedi, Manling Li, Heng Ji, Huan Zhang, Tong Zhang
ICML 2025 (Oral, Top 1%)

Embodied AgentsMultimodal LLMsEmbodied AI BenchmarkVision-Driven Agents

Why Is Spatial Reasoning Hard for VLMs? An Attention Mechanism Perspective on Focus Areas [PDF][Code][Data]
Shiqi Chen, Tongyao Zhu, Ruochen Zhou, Jinghan Zhang, Siyang Gao, Juan Carlos Niebles, Mor Geva, Junxian He, Jiajun Wu, Manling Li
ICML 2025

Spatial ReasoningAttention MechanismsVLM InterpretabilityAdaptive Decoding

Bring Reason to Vision: Understanding Perception and Reasoning through Model Merging [Website][PDF][Code][Data]
Shiqi Chen, Jinghan Zhang, Tongyao Zhu, Wei Liu, Siyang Gao, Miao Xiong, Manling Li, Junxian He
ICML 2025

Model MergingPerception and ReasoningVision-Language Models

T*: Re-thinking Temporal Search for Long-Form Video Understanding [Website][PDF][Data][Code]
Jinhui Ye*, Zihan Wang*, Haosen Sun, Keshigeyan Chandrasegaran, Zane Durante, Cristobal Eyzaguirre, Yonatan Bisk, Juan Carlos Niebles, Ehsan Adeli, Li Fei-Fei, Jiajun Wu, Manling Li
CVPR 2025, Oral at ICCV 2025 Workshop on Long Multi-Scene Video Foundations

Long-Form Video UnderstandingTemporal SearchVideo-Language ModelsKeyframe Selection

LayoutVLM: Differentiable Optimization of 3D Layout via Vision-Language Models [Website][PDF][Code]
Fan-Yun Sun, Weiyu Liu, Siyi Gu, Dylan Lim, Goutam Bhat, Federico Tombari, Manling Li, Nick Haber, Jiajun Wu
CVPR 2025

3D Scene LayoutSpatial ReasoningDifferentiable OptimizationVision-Language Models

Visually Descriptive Language Modeling for Vector Graphics Reasoning [PDF][Website][Code]
Zhenhailong Wang, Joy Hsu, Xingyao Wang, Kuan-Hao Huang, Manling Li, Jiajun Wu, Heng Ji
TMLR

Vector Graphics ReasoningVisual ReasoningSymbolic Representations

HourVideo: 1-Hour Video-Language Understanding [Website][PDF][Data][Code]
Keshigeyan Chandrasegaran, Agrim Gupta, Taran Kota, Lea M. Hadzic, Jimming He, Cristobal Eyzaguirre, Zane Durante, Manling Li, Jiajun Wu, Li Fei-Fei
NeurIPS 2024 Benchmark Track

Long-Form Video UnderstandingVideo-Language BenchmarkEgocentric Video

IKEA Manuals at Work: 4D Grounding of Assembly Instructions on Internet Videos [Website][PDF][Data][Code]
Yunong Liu, Weiyu Liu, Shubh Khanna, Cristobal Eyzaguirre, Manling Li, Juan Carlos Niebles, Vineeth Ravi, Saumitra Mishra, Jiajun Wu
NeurIPS 2024 Benchmark Track

4D GroundingAssembly InstructionsProcedural Video Understanding

Deep Concept Injection for Zero-shot Multimodal Reasoning [PDF]
Xudong Lin, Manling Li, Richard Zemel, Heng Ji, Shih-Fu Chang
EMNLP 2024

Zero-Shot Multimodal ReasoningConcept InjectionVideo Question Answering

ViStruct: Visual Structural Knowledge Extraction via Curriculum Guided Code-Vision Representation [PDF]
Yangyi Chen, Xingyao Wang, Manling Li, Derek Hoiem, Heng Ji
EMNLP 2023

Visual Knowledge ExtractionCurriculum LearningCode-Vision Representation

Towards Fast Adaptation of Pretrained Contrastive Models for Multi-channel Video-Language Retrieval [PDF] [Code]
Xudong Lin, Simran Tiwari, Shiyuan Huang, Manling Li, Mike Zheng Shou, Heng Ji, Shih-Fu Chang
CVPR 2023

Video-Language RetrievalContrastive LearningEfficient Adaptation

Open Visual Knowledge Extraction via Relation-Oriented Multimodality Model Prompting [PDF][Website]
Hejie Cui, Xinyu Fang, Zihan Zhang, Ran Xu, Xuan Kan, Xin Liu, Manling Li, Yangqiu Song, Carl Yang
NeurIPS 2023

Visual Knowledge ExtractionMultimodal PromptingRelation Extraction

Learning to Decompose Visual Features with Latent Textual Prompts [PDF] [Code]
Feng Wang^†, Manling Li, Xudong Lin, Hairong Lv, Alexander Schwing, Heng Ji
ICLR 2023 (^† denotes supervised undergraduate)

Visual Feature DecompositionPrompt LearningCLIP

Video Event Extraction via Tracking Visual States of Arguments [PDF] [Code]
Guang Yang^†, Manling Li, Jiajie Zhang, Xudong Lin, Shih-Fu Chang, Heng Ji
AAAI 2023 (^† denotes supervised undergraduate)

Video Event ExtractionVisual State TrackingMultimodal Information Extraction

Language Models with Image Descriptors are Strong Few-Shot Video-Language Learners [PDF] [Code]
Zhenhailong Wang^†*,Manling Li*, Ruochen Xu, Luowei Zhou, Jie Lei, Xudong Lin, Shuohang Wang, Ziyi Yang, Chenguang Zhu, Derek Hoiem, Shih-Fu Chang, Mohit Bansal, Heng Ji
NeurIPS'22 (equal contribution)

Few-Shot Video-Language LearningImage DescriptorsLLM Prompting

CLIP-Event:Connecting Vision and Text with Event Structures [PDF] [Data] [Code]
Manling Li, Ruochen Xu, Shuohang Wang, Xudong Lin, Chenguang Zhu, Xuedong Huang, Heng Ji, Shih-Fu Chang
CVPR'22 (Oral, Top 4.1%)

Event StructuresVision-Language PretrainingZero-Shot TransferContrastive Learning

Joint Multimedia Event Extraction from Video and Article [PDF] [Data]
Brian Chen, Xudong Lin, Christopher Thomas, Manling Li, Shoya Yoshida, Lovish Chum, Heng Ji and Shih-Fu Chang
EMNLP'21 Findings

Multimedia Event ExtractionVideo-Text UnderstandingInformation Extraction

Cross-media Structured Common Space for Multimedia Event Extraction [PDF] [Code] [Slides]
Manling Li*, Alireza Zareian*, Qi Zeng, Spencer Whitehead, Di Lu, Heng Ji, Shih-Fu Chang
ACL'20: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp.2557–2568

Multimedia Event ExtractionCross-Media Common SpaceWeakly Supervised Learning