Hi, there! I’m Manling. I am an assistant professor at the Computer Science department of Northwestern University (full-time starting in Fall 2024) and a postdoc at Stanford Vision and Learning Lab working with Prof Jiajun Wu. I obtained my Ph.D. degree in Computer Science at University of Illinois Urbana-Champaign in 2023, working with Prof. Heng Ji as a member of the Blender NLP lab. At the core of my research in Knowledge Foundation Models, I aim to equip machines with factual knowledge extraction and reasoning from multimodal data (Language + X, where X can be images, videos, robotics, audio, etc). The ultimate goal of my research is to promote factuality and truthfulness in information access, through a structured knowledge view that is easily explainable, highly compositional, and capable of long-horizon reasoning.

Bio: Her work on multimodal knowledge extraction won the ACL'20 Best Demo Paper Award, and the work on scientific information extraction from COVID literature won NAACL'21 Best Demo Paper Award. She was a recipient of Microsoft Research PhD Fellowship in 2021. She was selected as a DARPA Riser in 2022 (nominated by DARPA), and a EE CS Rising Star in 2022. She was awarded C.L. Dave and Jane W.S. Liu Award, and has been selected as a Mavis Future Faculty Fellow. She led 19 students to develop the UIUC information extraction system and ranked 1st in DARPA AIDA evaluation each phase. She has more than 30 publications on multimodal knowledge extraction and reasoning, and gave tutorials about event-centric multimodal knowledge at ACL'21, AAAI'21, NAACL'22, AAAI'23, etc.

Prospective students: I have 2-5 PhD positions and 1 postdoc position in Fall 2024 or later. Please apply to NorthwesternCS, which is open until Dec 15. Please also submit this form. Due to the large amount of emails, I may miss some of them. The best way to reach me is by filling out the form.

Junior PhD/master/undergraduate students: I will dedicate 30 minutes each week to offer guidance/suggestions/mentorship, especially for students from underrepresented groups or whoever is in need. If you would like to chat about life, career path, graduate school applications, or research ideas related to AI/ML, feel free to file the form to schedule a meeting.

Interests

Research Summary

I study Multimodal Information Extraction to discover factual knowledge and pattern knowledge from open-world multimodal data. My recent work aims to

read complex situation, think and reason about temporal patterns, and write truthfully with grounded knowledge.

Read: Complex Multimodal Semantics

Read: Complex Multimodal Semantics

Challenge: Multimodal Semantic Structures
My Solution: Opens up a new direction of multimodel event extraction, to construct a multimodal common semantic space that preserves event semantic structures

CLIP-Event: A pioneer to introduce event semantics to vision-language pretraining; Supports zero-shot vision event extraction for the first time M2E2: A new task of multimedia event extraction with the first benchmark released GAIA: The first open-source event-centric multimedia event extraction system COVID-KG for Drug Repurposing: A knowledge graph extracted from literature, which has been downloaded 2000+ times since 2021
Think: Long-Horizon Dynamics

Think: Long-Horizon Dynamics

Challenge: Long-Horizon Temporal Dynamics
My Solution: Establishs a new paradigm of Models as Schemas to add predictive power to produce multiple hypotheses, along with structural justifications for participant-specific and attribute-specific connections

Event Graph Schema: Open doors to graph context for event dynamics Path Language Model: A new step towards deep semantic understanding of inter-event connections VidIL: Creates a new paradigm to use language models to understand videos; The event prediction performance without any training videos is higher than models pretrained on 20k+ videos
Write: Truthfully

Write: Truthfully

Challenge: Document-, Corpus-, Web-level Analysis
My Solution: Creates a new document-/corpus-level representation based on event graphs to facilate large-scale tracking both temporally and geographically

Timeline: Hundreds of documents -> A short timeline COVID-19 Claim Radar: Open web news -> Supporting/refuting claims MuMuQA: Multimodal Multihop Question Answering Report Generation: Open web news -> Intelligence report Meeting Summarization: ASR transcripts -> A short summary

Core Publications

*
All Topic1-Read/Extraction Topic2-Think/Reasoning Topic3-Write/Generation
CLIP-Event: Connecting Vision and Text with Event Structures
Manling Li, Ruochen Xu, Shuohang Wang, Xudong Lin, Chenguang Zhu, Michael Zeng, Heng Ji, Shih-Fu Chang
- Open doors to introducing event semantic structures into vision-language understanding
- Bridges the gap between two modalities during V+L pretraining by optimizing structural alignment
Knowledge Extraction CVPR 2022 (Oral)
CLIP-Event: Connecting Vision and Text with Event Structures
Cross-media Structured Common Space for Multimedia Event Extraction
Manling Li*, Alireza Zareian*, Qi Zeng, Spencer Whitehead, Di Lu, Heng Ji, Shih-Fu Chang
- Opens up a new research direction of event-centric multimedia information extraction
- Multimedia knowledge extraction is a new paradigm for information extraction, and has been well-noticed by the research community
Knowledge Extraction ACL 2020
Cross-media Structured Common Space for Multimedia Event Extraction
GAIA: A Fine-grained Multimedia Knowledge Extraction System
Manling Li, Alireza Zareian*, Ying Lin, Xiaoman Pan, Spencer Whitehead, Brian Chen, Bo Wu, Heng Ji, Shih-Fu Chang, Clare R Voss, Dan Napierski, Marjorie Freedman
- The first open-source multimedia knowledge extraction system
- Used by various government agencies (e.g., ARL, DARPA, and IARPA)
- A top performer at the DARPA AIDA/NIST SM-KBP evaluation in each phase
- Received the ACL 2020 Best Demo Paper Award
Knowledge Extraction ACL 2020 Best Demo Paper Ranked 1st in DARPA AIDA evaluation (TA1) each phase since 2019 Transition to ARL and AFRL DARPA Demo Day
GAIA: A Fine-grained Multimedia Knowledge Extraction System
COVID-19 Literature Knowledge Graph Construction and Drug Repurposing Report Generation
Qingyun Wang, Manling Li, Xuan Wang, Nikolaus Parulian, Guangxing Han, Jiawei Ma, Jingxuan Tu, Ying Lin, Haoran Zhang, Weili Liu, Aabhas Chauhan, Yingjun Guan, Bangzheng Li, Ruisong Li, Xiangchen Song, Heng Ji, Jiawei Han, Shih-Fu Chang, James Pustejovsky, David Liem, Ahmed Elsayed, Martha Palmer, Jasmine Rah, Clare Voss, Cynthia Schneider, Boyan Onyshkevych
- A graph of genes, diseases, symptoms, chemicals, as well as their connections
- Awarded NAACL 2021 Best Demo Paper
- Generate a drug re-purposing report during collaborations with Prof David Liem from UCLA Data Science in Cardiovascular Medicine
Knowledge Extraction NAACL 2021 Best Demo Paper The Knowledge Base has been download 2000+ times since 2021
COVID-19 Literature Knowledge Graph Construction and Drug Repurposing Report Generation
The Future is not One-dimensional: Complex Event Schema Induction via Graph Modeling
Manling Li, Sha Li, Zhenhailong Wang, Lifu Huang, Kyunghyun Cho, Heng Ji, Jiawei Han
- Can input partially instantiated graphs to ``grow’’ the graph either forward or backward in time to predict missing events, arguments or relations, both from the past and in the future
- A different perspective, Models as Schemas, to output probabilistic schema models that can be probed on demand for event prediction
- A complicated graph including temporal dynamics and multiple paths involving entities that play important roles in a cohesive narrative - The first to capture event temporal dynamics probabilistically and perform global inference for event prediction
Knowledge Reasoning EMNLP 2021
The Future is not One-dimensional: Complex Event Schema Induction via Graph Modeling
Connecting the Dots: Event Graph Schema Induction with Path Language Modeling
Manling Li, Qi Zeng, Ying Lin, Kyunghyun Cho, Heng Ji, Jonathan May, Nathanael Chambers, Clare Voss
- An auto-regressive language model trained on event-event paths, select paths showing how one event walks to another through entities
- Launchs a new step towards deep understanding
Knowledge Reasoning EMNLP 2020
Connecting the Dots: Event Graph Schema Induction with Path Language Modeling
Language Models with Image Descriptors are Strong Few-Shot Video-Language Learners
Zhenhailong Wang*, Manling Li*, Ruochen Xu, Luowei Zhou, Jie Lei, Xudong Lin, Shuohang Wang, Ziyi Yang, Chenguang Zhu, Derek Hoiem, Shih-Fu Chang, Mohit Bansal, Heng Ji (* denotes equal contribution)
- Starts a new paradigm of leveraging langauge models to understand videos
Knowledge Reasoning NeurIPS 2022
Language Models with Image Descriptors are Strong Few-Shot Video-Language Learners
Timeline Summarization based on Event Graph Compression via Time-Aware Optimal Transport
Manling Li, Tengfei Ma, Mo Yu, Lingfei Wu, Tian Guo, Heng Ji, Kathleen McKeown
- Creates a new document-/corpus-level representation based on event graphs, and successfully supports zero-shot timeline summarization via graph compression.
Knowledge-Driven Application EMNLP 2021
Timeline Summarization based on Event Graph Compression via Time-Aware Optimal Transport
Keep Meeting Summaries on Topic: Abstractive Multi-Modal Meeting Summarization
Manling Li, Lingyu Zhang, Heng Ji, Rich Radke
- Leverages agenda-based topics to segment meeting transcripts
- The first to take advantage of multi-modal sensing of the meeting environment, such as cameras to capture each participant’s head pose and eye gaze.
Knowledge-Driven Applications ACL 2019
Keep Meeting Summaries on Topic: Abstractive Multi-Modal Meeting Summarization
COVID-19 Claim Radar: A Structured Claim Extraction and Tracking System
Manling Li, Revanth Gangi Reddy, Ziqi Wang, Yi-Shyuan Chiang, Tuan M. Lai, Pengfei Yu, Zixuan Zhang, Heng Ji
- Provides claims with claimer and claimer epistemic status (refuting/supporting) for the first time, placing knowledge within a social context
Knowledge-Driven Application ACL 2022 (Demo)
COVID-19 Claim Radar: A Structured Claim Extraction and Tracking System

News

  • [2023-10] Will organize “Towards Knowledgeable Language Models @ ACL24 Workshop” Call for Papers, submission deadline in May 20, 2024.
  • [2023-08] Joined Stanford Vision and Learning lab as a postdoc working with Prof Jiajun Wu. Excited!
  • [2023-07] Served as a session chair for multimodality at ACL 2023.
  • [2023-06] Will serve as an Area Chair at EMNLP 2023.
  • [2023-05] Will join Northwestern Computer Science as a tenure-track assistant professor.
  • [2022-12] Will serve as an Area Chair at ACL 2023.
  • [2022-10] Will present at EE CS Rising Star workshop in UT Austin on Oct 25-26 as a Riser.
  • [2022-10] Will present at DARPA Forward event in OSU on Oct 4-5 as a DARPA Riser.
  • [2022-07] Will give a tutorial at NAACL T3: New Frontiers of Information Extraction at 9:00–12:30 on July 10th (Location: Columbia D) with Muhao Chen, Lifu Huang, Ben Zhou, Heng Ji and Dan Roth. Welcome to our tutorial!

Full Publications

Please find my latest publications on Google Scholar.

Our workshop is calling for papers! Towards Knowledgeable Language Models @ ACL 2024 Workshop [Website]

Papers (* denotes equal contribution, † denotes mentored students)

2023


LM-Switch: Flexible Language Model Conditioning [PDF]
Chi Han, Jialiang Xu, Manling Li, Yi Fung, Chenkai Sun, Nan Jiang, Tarek Abdelzaher, Heng Ji
arXiv

Deep Concept Injection for Zero-shot Multimodal Reasoning [PDF]
Xudong Lin, Manling Li, Richard Zemel, Heng Ji, Shih-Fu Chang
arXiv

Rethinking and Controlling Object Existence Hallucinations in Large Vision Language Models [PDF]
Bohan Zhai, Shijia Yang, Xiangchen Zhao, Chenfeng Xu, Sheng Shen, Dongdi Zhao, Kurt Keutzer, Manling Li, Tan Yan, Xiangjun Fan
arXiv

ViStruct: Visual Structural Knowledge Extraction via Curriculum Guided Code-Vision Representation [PDF]
Yangyi Chen, Xingyao Wang, Manling Li, Derek Hoiem, Heng Ji
EMNLP 2023

Defining a New NLP Playground [PDF]
Sha Li, Chi Han, Pengfei Yu, Carl Edwards, Manling Li, Xingyao Wang, Yi Fung, Charles Yu, Joel R. Tetreault, Eduard H Hovy, Heng Ji
EMNLP 2023 Findings

Knowledge-Driven Vision-Language Encoding [Website]
Manling Li, Xudong Lin, Jie Lei, Mohit Bansal, Carl Vondrick, Shih-Fu Chang, Heng Ji
CVPR 2023: Tutorial

Towards Fast Adaptation of Pretrained Contrastive Models for Multi-channel Video-Language Retrieval [PDF] [Code]
Xudong Lin, Simran Tiwari, Shiyuan Huang, Manling Li, Mike Zheng Shou, Heng Ji, Shih-Fu Chang
CVPR 2023

Open Visual Knowledge Extraction via Relation-Oriented Multimodality Model Prompting [Website]
Hejie Cui, Xinyu Fang, Zihan Zhang, Ran Xu, Xuan Kan, Xin Liu, Manling Li, Yangqiu Song, Carl Yang
NeurIPS 2023

Non-Sequential Graph Script Induction via Multimedia Grounding [PDF]
Yu Zhou†, Sha Li, Manling Li, Xudong Lin, Shih-Fu Chang, Mohit Bansal and Heng Ji
ACL 2023 († denotes supervised undergraduate)

A Language First Approach to Procedure Planning [PDF]
Jiateng Liu†, Sha Li, Zhenhailong Wang, Manling Li, Heng Ji
ACL 2023 Findings († denotes supervised undergraduate)

Open-Domain Hierarchical Event Schema Induction by Incremental Prompting and Verification [PDF]
Sha Li, Ruining Zhao†, Manling Li, Heng Ji, Chris Callison-Burch and Jiawei Han
ACL 2023 († denotes supervised undergraduate)

Multimedia Generative Script Learning for Task Planning [PDF]
Qingyun Wang, Manling Li, Hou Pong Chan, Lifu Huang, Julia Hockenmaier, Girish Chowdhary and Heng Ji
ACL 2023 Findings

Learning to Decompose Visual Features with Latent Textual Prompts [PDF] [Code]
Feng Wang†, Manling Li, Xudong Lin, Hairong Lv, Alexander Schwing, Heng Ji
ICLR 2023 († denotes supervised undergraduate)

Knowledge-Driven Vision-Language Pretraining [PDF] [Website]
Manling Li, Xudong Lin, Jie Lei, Mohit Bansal, Shih-Fu Chang, Heng Ji
AAAI 2023: Tutorial

Video Event Extraction via Tracking Visual States of Arguments [PDF] [Code]
Guang Yang†, Manling Li, Jiajie Zhang, Xudong Lin, Shih-Fu Chang, Heng Ji
AAAI 2023 († denotes supervised undergraduate)

ADEPT: A DEbiasing PrompT Framework [PDF] [Code]
Ke Yang†, Charles Yu, Yi Fung, Manling Li, Heng Ji
AAAI 2023 († denotes supervised undergraduate)


2022


Language Models with Image Descriptors are Strong Few-Shot Video-Language Learners [PDF] [Code]
Zhenhailong Wang†*,Manling Li*, Ruochen Xu, Luowei Zhou, Jie Lei, Xudong Lin, Shuohang Wang, Ziyi Yang, Chenguang Zhu, Derek Hoiem, Shih-Fu Chang, Mohit Bansal, Heng Ji
NeurIPS'22 (equal contribution)

CLIP-Event:Connecting Vision and Text with Event Structures [PDF] [Data] [Code]
Manling Li, Ruochen Xu, Shuohang Wang, Xudong Lin, Chenguang Zhu, Xuedong Huang, Heng Ji, Shih-Fu Chang
CVPR'22 (Oral)

COVID-19 Claim Radar: A Structured Claim Extraction and Tracking System [PDF] [Code] [Demo] [Video]
Manling Li, Revanth Gangi Reddy, Ziqi Wang, Yi-Shyuan Chiang, Tuan M. Lai, Pengfei Yu, Zixuan Zhang,Heng Ji
ACL'22 Demo

Event Schema Induction with Double Graph Autoencoders [PDF] [Code]
Xiaomeng Jin†, Manling Li and Heng Ji
NAACL'22

New Frontiers of Information Extraction [PDF] [Website] [Slides] [Videos]
Muhao Chen, Lifu Huang, Manling Li, Ben Zhou, Heng Ji
NAACL'22: Tutorial

MuMuQA: Multimedia Multi-Hop News Question Answering via Cross-Media Knowledge Extraction and Grounding [PDF] [Data]
Revanth Gangi Reddy†, Xilin Rui†, Manling Li, Xudong Lin, Haoyang Wen, Jaemin Cho, Lifu Huang, Mohit Bansal, Avi Sil, Shih-Fu Chang, Alexander Schwing, Heng Ji
AAAI'22


2021


The Future is not One-dimensional: Complex Event Schema Induction by Graph Modeling for Event Prediction [PDF] [Data]
Manling Li, Sha Li, Zhenhailong Wang, Lifu Huang, Kyunghyun Cho, Heng Ji, Jiawei Han and Clare Voss
EMNLP'21

Timeline Summarization based on Event Graph Compression via Time-Aware Optimal Transport [PDF] [Data]
Manling Li, Tengfei Ma, Mo Yu, Lingfei Wu, Tian Gao, Heng Ji and Kathleen McKeown
EMNLP'21

Joint Multimedia Event Extraction from Video and Article [PDF] [Data]
Brian Chen, Xudong Lin, Christopher Thomas, Manling Li, Shoya Yoshida, Lovish Chum, Heng Ji and Shih-Fu Chang
EMNLP'21 Findings

Event-centric Natural Language Processing [PDF] [Slides]
Muhao Chen, Hongming Zhang, Qiang Ning, Manling Li, Heng Ji, Kathleen McKeown and Dan Roth
ACL'21: Tutorial.

GENE: Global Event Network Embedding [PDF]
Qi Zeng, Manling Li, Tuan Lai, Heng Ji, Mohit Bansal and Hanghang Tong
NAACL'21: TextGraphs 2021 Workshop

COVID-19 Literature Knowledge Graph Construction and Drug Repurposing Report Generation [PDF] [Code/Data]
Qingyun Wang, Manling Li, Xuan Wang, Nikolaus Parulian, Guangxing Han, Jiawei Ma, Jingxuan Tu, Ying Lin, Haoran Zhang, Weili Liu, Aabhas Chauhan, Yingjun Guan, Bangzheng Li, Ruisong Li, Xiangchen Song, Heng Ji, Jiawei Han, Shih-Fu Chang, James Pustejovsky, David Liem, Ahmed Elsayed, Martha Palmer, Jasmine Rah, Clare Voss, Cynthia Schneider, Boyan Onyshkevych
NAACL'21: System Demonstrations
(Best Demo Paper Award at NAACL2021)

RESIN: A Dockerlized Schema-Guided Cross-document Cross-lingual Cross-media Information Extraction and Event Tracking System [PDF] [Code]
Haoyang Wen, Ying Lin, Tuan M. Lai, Xiaoman Pan, Sha Li, Xudong Lin, Ben Zhou, Manling Li, Haoyu Wang, Hongming Zhang, Xiaodong Yu, Alexander Dong, Zhenhailong Wang, Yi R. Fung, Piyush Mishra, Qing Lyu, Dídac Surís, Brian Chen, Susan W. Brown, Martha Palmer, Chris Callison-Burch, Carl Vondrick, Jiawei Han, Dan Roth, Shih-Fu Chang and Heng Ji
NAACL'21: System Demonstrations

Event-centric Natural Language Processing [PDF] [Slides]
Muhao Chen, Hongming Zhang, Qiang Ning, Manling Li, Heng Ji and Dan Roth
AAAI'21: Tutorial.


2020


Connecting the Dots: Event Graph Schema Induction with Path Language Modeling [PDF] [Code/Data] [Slides]
Manling Li, Qi Zeng, Ying Lin, Kyunghyun Cho, Heng Ji, Jonathan May, Nathanael Chambers and Clare Voss
EMNLP'20: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing.

GAIA: A Fine-grained Multimedia Knowledge Extraction System [PDF] [Code] [Video]
Manling Li*, Alireza Zareian*, Ying Lin, Xiaoman Pan, Spencer Whitehead, Brian Chen, Bo Wu, Heng Ji, Shih-Fu Chang, Clare R. Voss, Dan Napierski, Marjorie Freedman
ACL'20: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations. pp. 77–86
(Best Demo Paper Award at ACL2020)

Cross-media Structured Common Space for Multimedia Event Extraction [PDF] [Code] [Slides]
Manling Li*, Alireza Zareian*, Qi Zeng, Spencer Whitehead, Di Lu, Heng Ji, Shih-Fu Chang
ACL'20: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp.2557–2568

GAIA at SM-KBP 2020 - A Dockerized Multi-media Multi-lingual Knowledge Extraction, Clustering, Temporal Tracking and Hypothesis Generation System [PDF] [Project]
Manling Li, Ying Lin, Tuan Manh Lai, Xiaoman Pan, Haoyang Wen, Sha Li, etc %Zhenhailong Wang, Pengfei Yu, Lifu Huang, Di Lu, Qingyun Wang, Haoran Zhang, Qi Zeng, Chi Han, Zixuan Zhang, Yujia Qin, Xiaodan Hu, Nikolaus Parulian, Daniel Campos, Heng Ji, Brian Chen, Xudong Lin, Alireza Zareian, Amith Ananthram, Emily Allaway, Shih-Fu Chang, Kathleen McKeown, Yixiang Yao, Yifan Wang, Michael Spector, Mitchell DeHaven, Daniel Napierski, Marjorie Freedman, Pedro Szekely, Haidong Zhu, Ram Nevatia, Yang Bai, Yifan Wang, Ali Sadeghian, Haodi Ma, Daisy Zhe Wang
TAC-KBP: Text Analysis Conference Knowledge Base Population Workshop 2020 (Rank 1st in the leaderboard.)

UIUC TAC2020 RUFES System Description [PDF] [Project]
Revanth Gangi Reddy, Manling Li, Haoyang Wen and Heng Ji
TAC-KBP: Text Analysis Conference Knowledge Base Population Workshop 2020


2019


Keep Meeting Summaries on Topic: Abstractive Multi-Modal Meeting Summarization
[PDF] [Code]
Manling Li, Lingyu Zhang, Heng Ji, Rich Radke
ACL'19: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp.2190–2196

Multilingual Entity, Relation, Event and Human Value Extraction [PDF] [Code] [Video]
Manling Li, Ying Lin, Joe Hoover, Spencer Whitehead, Clare Voss, Morteza Dehghani, Heng Ji
NAACL'19: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (Demonstrations), pp.110–115

The Unobtrusive Group Interaction (UGI) Corpus [PDF] [Code]
Indrani Bhattacharya, Michael Foley, Ni Zhang, Tongtao Zhang, Christine Ku, Cameron Mine, Manling Li, Heng Ji, etc.
ACM MMSys'19: ACM Multimedia Systems Conference 2019, pp.249-254

GAIA at SM-KBP 2019 - A Multi-media Multi-lingual KnowledgeExtraction and Hypothesis Generation System [PDF] [Project]
Manling Li, Ying Lin, Ananya Subburathinam, Spencer Whitehead, Xiaoman Pan, Di Lu, Qingyun Wang, Tongtao Zhang, Lifu Huang, Heng Ji. etc
TAC-KBP: Text Analysis Conference Knowledge Base Population Workshop 2019 (Rank 1st, with more than 10% higher than the second team.)

A Baseline Fine-Grained Entity Extraction System for TAC-KBP2019 [PDF] [Project]
Ying Lin, Xiaoman Pan, Manling Li, Heng Ji
TAC-KBP: Text Analysis Conference Knowledge Base Population Workshop 2019


2018 and Before


Link Prediction in Knowledge Graphs: A Hierarchy-Constrained Approach [PDF] [Code]
Manling Li, Yuanzhuo Wang, Denghui Zhang, Yantao Jia, Xueqi Cheng
IEEE Transactions on Big Data, pp.1-14

Hierarchical Types Constrained Topic Entity Detection for Knowledge Base Question Answering [PDF]
Yunqi Qiu, Manling Li, Yuanzhuo Wang, Yantao Jia, Xiaolong Jin, Xueqi Cheng
WWW 2018: Companion Proceedings of The Web Conference 2018, pp.35-36 (abstract paper)

Path-Based Attention Neural Model for Fine-Grained Entity Typing [PDF] [Code]
Denghui Zhang, Manling Li, Pengshan Cai, Yantao Jia, Yuanzhuo Wang, Xueqi Cheng
AAAI 2018: Proceedings of the 32nd AAAI Conference on Artificial Intelligence, pp.8179-8180 (abstract paper)

Efficient Parallel Translating Embedding For Knowledge Graphs [PDF] [Code]
Denghui Zhang, Manling Li, Yantao Jia, Yuanzhuo Wang, Xueqi Cheng
WI 2017: Proceedings of IEEE/WIC/ACM International Conference on Web Intelligence, pp.460-468

OpenKN at TAC KBP 2016 [PDF] [Project]
Manling Li, Xinlei Chen, Yantao Jia, Yuanzhuo Wang, etc.
TAC-KBP: Text Analysis Conference Knowledge Base Population Workshop 2016

Hierarchy-Based Link Prediction in Knowledge Graphs [PDF] [Poster]
Manling Li, Yantao Jia, Yuanzhuo Wang, Jingyuan Li, Xueqi Cheng.
WWW 2016: Proceedings of the 25th International Conference Companion on World Wide Web, pp.77-78 (abstract paper)

Predicting Links and Their Building Time: A Path-Based Method [PDF] [Poster]
Manling Li, Yantao Jia, Yuanzhuo Wang, Zeya Zhao, Xueqi Cheng.
AAAI 2016: Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence, pp.4228-4229 (abstract paper)

Patents

Patents


6. Knowledge Graph Compression
Tengfei Ma, Tengfei Ma, Mo Yu, Lingfei Wu, Tian Gao, Manling Li. (1st student author).
US Patent (submitted, under IBM Research)

5. A Tag Inference System Based on Knowledge Graph Embedding [PDF]
Xueqi Cheng, Yantao Jia, Pengshan Cai, Manling Li, etc. (2nd student author).
No.CN201710469315.4.

4. An Ontology Alignment Algorithm Based on Knowledge Graph Embedding [PDF]
Xueqi Cheng, Yantao Jia, Manling Li, etc. (1st student author).
No.CN201710230135.0.


3. A Link Prediction System Based on Knowledge Graph Embedding [PDF]
Xueqi Cheng, Yantao Jia, Manling Li, etc. (1st student author).
No.CN201710041593.X.

2. A Fine-Grained Entity Typing Method for Knowledge Base Population [PDF]
Xueqi Cheng, Yantao Jia, Hailun Lin, Manling Li, etc. (2nd student author).
No.CN201510033050.4.


1. An Attribute Extraction Method for Web Pages [PDF]
Xueqi Cheng, Yantao Jia, Zeya Zhao, Manling Li, etc. (2nd student author).
No.CN201510071993.6.

Talks

Tutorials

Invited Talks

  • From Entity-Centric to Event-Centric Multimodal Event Knowledge Acquisition
    EE CS Rising Star Workshop, University of Texas at Austin, USA. Oct 2022.

  • Towards Accurate Intelligent Analysis: Event-Centric Multimedia Knowledge Extraction
    DARPA Forward (Invite-Only), USA. Oct 2022.

  • Event-Centric Multimedia Data Understanding
    Ohio State University, USA. Oct 2022.
    Singapore Management University, Singapore. Oct 2022.
    George Mason University, USA. Oct 2022.
    North Carolina State University, USA. Oct 2022.

  • Multimedia Event Extraction: From Object-Centric to Event-Centric
    Virginia Tech, USA. Sept 2022.

  • Event Knowledge Graph Construction
    LOGS Graph Reasoning Seminar. Aug 2022.

  • Event Graph Structures in Vision-Language Understanding
    DataFun. Jun 2022.

  • Connecting Vision and Text using Event Structures
    NewsBreak. Apr 2022.

  • Memories as Repositories of Events: Structural Event Knowledge Acquisition
    University of Notre Dame. Feb 2022.

  • Comprehensive Event Understanding in Multimedia Data
    USC ISI. Dec 2021.

  • Structural Event Knowledge Acquisition from Multimedia Data
    UIUC NLP Seminar. Nov 2021.

  • Event Extraction and Reasoning in Multimedia News Data
    Microsoft Research. Nov 2021.

  • Improving Visual Event and Argument Role Understanding with Contrastive Image-Language Pretraining
    Microsoft Research. Aug 2021.

  • Fine-Grained Knowledge Extraction System from Multimedia Data
    ai.science. Oct 2020.

  • Event Understanding and Narration for Multimedia Data
    Intel MDI Research Lab. May 2020.

Academic Service

Area Chair:

  • ACL, 2023
  • EMNLP, 2023

Senior Program Committee:

  • IJCAI, 2021

Program Committee:

  • ARR (ACL Rolling Review, from 2021)
  • ACL (from 2021)
  • EMNLP (from 2021)
  • NAACL-HLT (from 2021)
  • AAAI (from 2021)
  • WWW (from 2021)
  • AKBC (2021)
  • EACL (from 2021)
  • KDD DI Workshop (2021)
  • NLPCC (2021)
  • COLING (from 2020)
  • AACL (from 2020)
  • CCL (from 2020)

Journal Reviewer:

  • IEEE MultiMedia
  • Journal of Artificial Intelligence Research
  • Journal of Advances in Information Technology

Community Services:

  • ACM Mentor, ACM Mentorship Program at UIUC, 2022
  • CS Ambassador, UIUC CS Visit Day for Prospective Graduate Students, 2022
  • Advising Assistant, UIUC PhD Orientation Seminar, 2021
  • Graduate Student Representative, UIUC CS Visit Day, 2020
  • Co-Chair, “Source” Department Technology Association in University of Science and Technology Beijing, 2014

Awards

Awards

ACL 2020 Best Demo Paper Award
NAACL 2021 Best Demo Paper Award
DARPA Riser 2022 (nominated by DARPA)
Microsoft Research PhD Fellowship 2021
EE CS Rising Star 2022
Mavis Future Faculty Fellow
C.L. and Jane Liu Award
National Scholarship
Outstanding Undergraduate Award in Beijing
Chinese Academy of Sciences First-Class Scholarship
Schlumberger Scholarship

Academic and Scientific Competitions

Ranked 1st in TAC Streaming Multimedia Knowledge Base Population (SM-KBP) 2020
Ranked 1st in TAC Streaming Multimedia Knowledge Base Population (SM-KBP) 2019, with more than 10% absolute gains compared the second ranked team

1st prize in National Computer Game Tournament
1st prize in National Software Innovation Competition
2nd prize in National Computer Design Competition
2nd prize in Student Research Training Program of China

1st prize in Beijing Micromouse Competition
2nd prize in Beijing Physics Experiment Competition

Teaching

Teaching Assistant:

  • CS 598 Knowledge-driven Natural Language Generation, Spring 2022

Advising Assistant:

  • CS 591 PhD Orientation Seminar, Fall 2021

Guest Lecturer:

  • Event-Centric Multimedia Encoding
    CS 6604 Advanced Topics in Natural Language Processing
    Virginia Tech, Fall 2022
  • Event-Centric Multimedia Encoding
    CSC 791&591: Advanced Topics in Efficient Deep Learning
    North Carolina State University, Fall 2022
  • Knowledge-Driven Vision-Language Pretraining
    CS 546 Advanced Topics in Natural Language Processing
    UIUC, Fall 2022
  • Recent Advances in Multimedia Encoding
    CS 546 Advanced Topics in Natural Language Processing
    UIUC, Fall 2022
  • Timeline Summarization: Introducing Temporal Dimensions into Summarization
    CS 598 Knowledge Driven Natural Language Generation
    UIUC, Spring 2022
  • Multimedia Encoding via Vision-Language Pretraining
    CS 546 Advanced Topics in Natural Language Processing
    UIUC, Fall 2021

Mentored Students

It is a great pleasure to work with such talented young people.
I am grateful for the trust that they have placed in me and for the support that I have received from my advisor, Prof Heng Ji.

Masters

Avatar

Feng Wang

ECE Master @ Tsinghua

Submitted a first-author paper to ICLR

Vision-Language Pretraining

Avatar

Zhenhailong Wang

CS Master @ UIUC

Published his first paper on schema induction at EMNLP 2021, and later a first-author paper at NeurIPS 2022

Vision-Language Pretraining

Undergraduates

Avatar

Guang Yang

CS Undergraduate @ Tsinghua

Published his first paper (first-author, AAAI 23)

Video Event Extraction

Avatar

Jialiang Xu

ECE Undergraduate @ UIUC

Will submit a first-author paper to EMNLP

Vision-Language Pretraining

Avatar

Jiateng Liu

CS Undergraduate @ Zhejiang University

Submitted his first paper to ACL 2023

Multimodal Schema Induction

Avatar

Ke Yang

ECE Undergraduate @ Tsinghua

Published his first paper (first-author, AAAI 23)

Language Model Debiasing

Avatar

Ruining Zhao

CS Undergraduate @ UIUC

Submitted her first paper to ACL 2023

Schema Induction

Avatar

Stuti Agrawal

CS Undergraduate @ UIUC

Schema Induction

Avatar

Yu (Bryan) Zhou

CS Undergraduate @ UCLA

Submitted his first paper to ACL (first-author)

Multimodal Schema Induction

Alumni

Avatar

Emma Hamel

CS Undergraduate @ UIUC

Worked on schema induction

Schema Induction

Avatar

Revanth Gangi Reddy

CS Master at UIUC (now PhD at UIUC)

Published a first-author paper at AAAI 2022

Multimedia New Question Answering

Avatar

Xilin Rui

CS Undergraduate @ Tsinghua

Published her first paper at AAAI 22

Multimedia Question Answering

Avatar

Zhiying Zhu

CS Undergraduate @ Zhejiang University

Now CS Master @ CMU

Vision Event Knowledge