Research

For a complete list of publications, please see the Google Scholar page.

2024

  1. FlexAttention for Efficient High-Resolution Vision-Language Models
    Junyan Li, Delin Chen, Tianle Cai , Peihao Chen, Yining Hong , Zhenfang Chen, Yikang Shen, and Chuang Gan
  2. RoboGen: Towards Unleashing Infinite Data for Automated Robot Learning via Generative Simulation
    Yufei Wang, Zhou Xian , Feng Chen , Tsun-Hsuan Wang , Yian Wang, Katerina Fragkiadaki, Zackory Erickson, David Held, and Chuang Gan
    arXiv PDF Code Website
  3. CoNav: A Benchmark for Human-Centered Collaborative Navigation
    Changhao Li, Xinyu Sun , Peihao Chen, Jugang Fan , Zixu Wang, Yanxia Liu, Jinhui Zhu, Chuang Gan, and Mingkui Tan
  4. COMBO: Compositional World Models for Embodied Multi-Agent Cooperation
    Hongxin Zhang , Zeyuan Wang, Qiushi Lyu , Zheyuan Zhang , Sunli Chen, Tianmin Shu, Yilun Du, and Chuang Gan
  5. SALMON: Self-Alignment with Instructable Reward Models
    Zhiqing Sun, Yikang Shen, Hongxin Zhang , Qinhong Zhou , Zhenfang Chen, David Cox , Yiming Yang, and Chuang Gan
  6. Thin-Shell Object Manipulations With Differentiable Physics Simulations
    Yian Wang , Juntian Zheng , Zhehuan Chen, Zhou Xian , Gu Zhang , Chao Liu, and Chuang Gan
  7. Visual Chain-of-Thought Prompting for Knowledge-Based Visual Reasoning
    Zhenfang Chen , Qinhong Zhou, Yikang Shen, Yining Hong , Zhiqing Sun, Dan Gutfreund, and Chuang Gan
    In AAAI Conference on Artificial Intelligence
  8. 3D-VLA: A 3D Vision-Language-Action Generative World Model
    Haoyu Zhen, Xiaowen Qiu , Peihao Chen, Jincheng Yang, Xin Yan, Yilun Du, Yining Hong, and Chuang Gan
    arXiv PDF Code Website
  9. Building Cooperative Embodied Agents Modularly with Large Language Models
    Hongxin Zhang , Weihua Du, Jiaming Shan , Qinhong Zhou, Yilun Du, Joshua B. Tenenbaum, Tianmin Shu, and Chuang Gan
  10. EfficientViT: Multi-Scale Linear Attention for High-Resolution Dense Prediction
    Han Cai, Junyan Li, Muyan Hu, Chuang Gan, and Song Han
  11. HAZARD Challenge: Embodied Decision Making in Dynamically Changing Environments
    Qinhong Zhou , Sunli Chen , Yisong Wang, Haozhe Xu , Weihua Du, Hongxin Zhang, Yilun Du, Joshua B. Tenenbaum, and Chuang Gan
  12. MultiPLY: A Multisensory Object-Centric Embodied Large Language Model in 3D World
    Yining Hong , Zishuo Zheng , Peihao Chen , Yian Wang, Junyan Li, and Chuang Gan

2023

  1. DCIR: Dynamic Consistency Intrinsic Reward for Multi-Agent Reinforcement Learning
    Kunyang Lin , Yufeng Wang , Peihao Chen, Runhao Zeng , Siyuan Zhou, Mingkui Tan, and Chuang Gan
  2. Principle-Driven Self-Alignment of Language Models from Scratch with Minimal Human Supervision
    Zhiqing Sun, Yikang Shen , Qinhong Zhou, Hongxin Zhang , Zhenfang Chen, David Cox , Yiming Yang, and Chuang Gan
    Advances in Neural Information Processing Systems
    arXiv PDF Code Website
  3. CoVLM: Composing Visual Entities and Relationships in Large Language Models Via Communicative Decoding
    Junyan Li, Delin Chen, Yining Hong , Zhenfang Chen , Peihao Chen, Yikang Shen, and Chuang Gan
  4. A^2Nav: Action-Aware Zero-Shot Robot Navigation by Exploiting Vision-and-Language Ability of Foundation Models
    Peihao Chen, Xinyu Sun, Hongyan Zhi, Runhao Zeng , Thomas H. Li , Gaowen Liu, Mingkui Tan, and Chuang Gan
  5. Learning Vision-and-Language Navigation from YouTube Videos
    Kunyang Lin , Peihao Chen , Diwei Huang , Thomas H. Li, Mingkui Tan, and Chuang Gan
  6. 3D-LLM: Injecting the 3D World into Large Language Models
    Yining Hong, Haoyu Zhen , Peihao Chen, Shuhong Zheng, Yilun Du , Zhenfang Chen, and Chuang Gan
    Advances in Neural Information Processing Systems
    arXiv PDF Code Website
  7. Masked Motion Encoding for Self-Supervised Video Representation Learning
    Xinyu Sun , Peihao Chen , Liangwei Chen , Changhao Li , Thomas H. Li, Mingkui Tan, and Chuang Gan

2022

  1. Learning Active Camera for Multi-Object Navigation
    Peihao Chen, Dongyu Ji, Kunyang Lin , Weiwen Hu , Wenbing Huang , Thomas H. Li, Mingkui Tan, and Chuang Gan
  2. Weakly-Supervised Multi-Granularity Map Learning for Vision-and-Language Navigation
    Peihao Chen, Dongyu Ji, Kunyang Lin, Runhao Zeng , Thomas H. Li, Mingkui Tan, and Chuang Gan

2021

  1. RSPNet: Relative Speed Perception for Unsupervised Video Representation Learning
    Peihao Chen, Deng Huang, Dongliang He, Xiang Long, Runhao Zeng, Shilei Wen, Mingkui Tan, and Chuang Gan

2020

  1. Relation Attention for Temporal Action Localization
    Peihao Chen, Chuang Gan , Guangyao Shen , Wenbing Huang, Runhao Zeng, and Mingkui Tan
    IEEE Transactions on Multimedia
  2. Location-aware Graph Convolutional Networks for Video Question Answering
    Deng Huang , Peihao Chen, Runhao Zeng , Qing Du, Mingkui Tan, and Chuang Gan
  3. Generating Visually Aligned Sound From Videos
    Peihao Chen , Yang Zhang, Mingkui Tan, Hongdong Xiao, Deng Huang, and Chuang Gan
    IEEE Transactions on Image Processing
  4. Foley Music: Learning to Generate Music from Videos
    Chuang Gan, Deng Huang , Peihao Chen, Joshua B. Tenenbaum, and Antonio Torralba
  5. Dense Regression Network for Video Grounding
    Runhao Zeng , Haoming Xu , Wenbing Huang , Peihao Chen, Mingkui Tan, and Chuang Gan

2019

  1. Breaking Winner-Takes-All: Iterative-Winners-Out Networks for Weakly Supervised Temporal Action Localization
    Runhao Zeng, Chuang Gan , Peihao Chen , Wenbing Huang, Qingyao Wu, and Mingkui Tan
    IEEE Transactions on Image Processing
  2. Self-supervised Moving Vehicle Tracking with Stereo Sound
    Chuang Gan, Hang Zhao , Peihao Chen, David Cox, and Antonio Torralba

2018

  1. soundofpixels.png
    The Sound of Pixels
    Hang Zhao, Chuang Gan, Andrew Rouditchenko, Carl Vondrick, Josh McDermott, and Antonio Torralba
    In The European Conference on Computer Vision (ECCV)
    arXiv PDF Code Website