Publications in 2024

UBSoft: A Simulation Platform for Robotic Skill Learning in Unbounded Soft Environments

Chunru Lin*, Jugang Fan*, Yian Wang, Zeyuan Yang, Zhehuan Chen, Lixing Fang, Tsun-Hsuan Wang, Zhou Xian, Chuang Gan

CoRL 2024

Paper

Project Page

ARCHITECT: Generating Vivid and Interactive 3D Scenes with Hierarchical 2D Inpainting

Yian Wang*, Xiaowen Qiu*, Jiageng Liu*, Zhehuan Chen, Jiting Cai, Tsun-Hsuan Wang, Yufei Wang, Zhou Xian, Chuang Gan

NeurIPS 2024

Paper

Project Page

Code

Physically Compatible 3D Object Modeling from a Single Image

Minghao Guo, Bohan Wang, Pingchuan Ma, Tianyuan Zhang, Crystal Elaine Owens, Chuang Gan, Joshua B. Tenenbaum, Kaiming He, Wojciech Matusik

NeurIPS 2024

Paper

Project Page

Code

ConMe: Rethinking Evaluation of Compositional Reasoning for Modern VLMs

Irene Huang, Wei Lin, M. Jehanzeb Mirza, Jacob A. Hansen, Sivan Doveh, Victor Ion Butoi, Roei Herzig, Assaf Arbelle, Hilde Kuehne, Trevor Darrell, Chuang Gan, Aude Oliva, Rogerio Feris, Leonid Karlinsky

NeurIPS 2024

Paper

Project Page

Code

Easy-to-Hard Generalization: Scalable Alignment Beyond Human Supervision

Zhiqing Sun, Longhui Yu, Yikang Shen, Weiyang Liu, Yiming Yang, Sean Welleck, Chuang Gan

NeurIPS 2024

Paper

Project Page

Code

SocialGPT: Prompting LLMs for Social Relation Reasoning via Greedy Segment Optimization

Wanhua Li*, Zibin Meng*, Jiawei Zhou, Donglai Wei, Chuang Gan, Hanspeter Pfister

NeurIPS 2024

Paper

Project Page

Code

Constrained Human-AI Cooperation: An Inclusive Embodied Social Intelligence Challenge

Weihua Du*, Qiushi Lyu*, Jiaming Shan, Zhenting Qi, Hongxin Zhang, Sunli Chen, Andi Peng, Tianmin Shu, Kwonjoon Lee, Behzad Dariush, Chuang Gan

NeurIPS 2024

Paper

Project Page

Code

FlexAttention for Efficient High-Resolution Vision-Language Models

Junyan Li, Delin Chen, Tianle Cai, Peihao Chen, Yining Hong, Zhenfang Chen, Yikang Shen, Chuang Gan

ECCV 2024

Paper

Project Page

Code

RoboGen: Towards Unleashing Infinite Data for Automated Robot Learning via Generative Simulation

Yufei Wang, Zhou Xian, Feng Chen, Tsun-Hsuan Wang, Yian Wang, Katerina Fragkiadaki, Zackory Erickson, David Held, Chuang Gan

ICML 2024

Paper

Project Page

Code

RoboDreamer: Learning Compositional World Models for Robot Imagination

Siyuan Zhou, Yilun Du, Jiaben Chen, Yandong Li, Dit-Yan Yeung, Chuang Gan

ICML 2024

Paper

Project Page

Code

Visual Chain-of-Thought Prompting for Knowledge-Based Visual Reasoning

Zhenfang Chen, Qinhong Zhou, Yikang Shen, Yining Hong, Zhiqing Sun, Dan Gutfreund, Chuang Gan

AAAI 2024

Paper

Code

3D-VLA: A 3D Vision-Language-Action Generative World Model

Haoyu Zhen, Xiaowen Qiu, Peihao Chen, Jincheng Yang, Xin Yan, Yilun Du, Yining Hong, Chuang Gan

ICML 2024

Paper

Code

LLM and Simulation as Bilevel Optimizers: A New Paradigm to Advance Physical Scientific Discovery

Pingchuan Ma, Johnson Tsun-Hsuan Wang, Minghao Guo, Zhiqing Sun, Josh Tenenbaum, Daniela Rus, Chuang Gan, Wojciech Matusik

ICML 2024

Paper

Project Page

Code

ContPhy: Continuum Physical Concept Learning and Reasoning from Videos

Zhicheng Zheng, Xin Yan, Zhenfang Chen, Jingzhou Wang, Qin Zhi Eddie Lim, Joshua Tenenbaum, Chuang Gan

ICML 2024

Paper

Project Page

Code

SALMON: Self-Alignment with Instructable Reward Models

Zhiqing Sun, Yikang Shen, Hongxin Zhang, Qinhong Zhou, Zhenfang Chen, David Cox, Yiming Yang, Chuang Gan

ICLR 2024

Paper

Code

Thin-Shell Object Manipulations With Differentiable Physics Simulations

Yian Wang, Juntian Zheng, Zhehuan Chen, Zhou Xian, Gu Zhang, Chao Liu, Chuang Gan

ICLR 2024 (Spotlight)

Paper

Project Page

Code

Building Cooperative Embodied Agents Modularly with Large Language Models

Hongxin Zhang, Weihua Du, Jiaming Shan, Qinhong Zhou, Yilun Du, Joshua B. Tenenbaum, Tianmin Shu, Chuang Gan

ICLR 2024

Paper

Project Page

HAZARD Challenge: Embodied Decision Making in Dynamically Changing Environments

Qinhong Zhou, Sunli Chen, Yisong Wang, Haozhe Xu, Weihua Du, Hongxin Zhang, Yilun Du, Joshua B. Tenenbaum, Chuang Gan

ICLR 2024

Paper

Project Page

Code

GENOME: GenerativE Neuro-symbOlic visual reasoning by growing and reusing ModulEs

Zhenfang Chen, Rui Sun, Wenjun Liu, Yining Hong, Chuang Gan

ICLR 2024

Paper

Project Page

Code

DiffTactile: A Physics-based Differentiable Tactile Simulator for Contact-rich Robotic Manipulation

Zilin Si, Gu Zhang, Qingwei Ben, Branden Romero, Zhou Xian, Chao Liu, Chuang Gan

ICLR 2024

Paper

Project Page

Code

CoVLM: Composing Visual Entities and Relationships in Large Language Models Via Communicative Decoding

Junyan Li, Delin Chen, Yining Hong, Zhenfang Chen, Peihao Chen, Yikang Shen, Chuang Gan

ICLR 2024

Paper

Project Page

Code

Open3DIS: Open-Vocabulary 3D Instance Segmentation with 2D Mask Guidance

Phuc D.A. Nguyen, Tuan Duc Ngo, Evangelos Kalogerakis, Chuang Gan, Cuong Pham, Khoi Nguyen

CVPR 2024

Paper

Project Page

SOK-Bench: A Situated Video Reasoning Benchmark with Aligned Open-World Knowledge

Andong Wang, Bo Wu, Sunli Chen, Zhenfang Chen, Wei-Ning Lee, Li Erran Li, Chuang Gan

CVPR 2024

Paper

Project Page

RILA: Reflective and Imaginative Language Agent for Zero-Shot Semantic Audio-Visual Navigation

Zeyuan Yang, Jiageng Liu, Peihao Chen, Anoop Cherian, Tim K Marks, Jonathan Le Roux, Chuang Gan

CVPR 2024

MultiPLY: A Multisensory Object-Centric Embodied Large Language Model in 3D World

Yining Hong, Zishuo Zheng, Peihao Chen, Yian Wang, Junyan Li, Chuang Gan

CVPR 2024

Paper

Code

ConceptGraphs: Open-Vocabulary 3D Scene Graphs for Perception and Planning

Qiao Gu, Alihusein Kuwajerwala, Sacha Morin, Krishna Murthy Jatavallabhula, Bipasha Sen, Aditya Agarwal, Corban Rivera, William Paul, Kirsty Ellis, Rama Chellappa, Chuang Gan, Celso Miguel de Melo, Joshua B. Tenenbaum, Antonio Torralba, Florian Shkurti, Liam Paull

ICRA 2024

Paper

Project Page

Code