Xiuming Zhang

About

Xiuming Zhang 張修明

I'm currently a Senior Machine Learning Scientist at Tesla Autopilot , working on 3D object and scene understanding for vehicle and robot autonomy. Before Tesla , I worked on computational photography at Adobe Emerging Products Group. I did my Ph.D. in computer vision and computer graphics with Bill Freeman at MIT CSAIL . During MIT , I also received mentorship from Jon Barron via internships at Google Research . My dissertation Shape, Reflectance, and Illumination From Appearance studies inverse rendering and its applications, including 3D shape reconstruction and free-viewpoint relighting. Here is a more formal, third-person bio.

Google Scholar xiuming6zhang at gmail dot com

Recent Experiences

Computer Scientist, Adobe Emerging Products Group , 2021–2022

Research Intern, Google Research , 2020–2021, w/ Jon Barron

Research Intern, Google Research , 2019–2020, w/ Yun-Ta Tsai & Jon Barron

Ph.D. & S.M. (CS), MIT CSAIL , 2016–2021, w/ Bill Freeman

News

Apr. 2023	DiffusionRig (personalized face editing) released for CVPR 2023.
Oct. 2022	Joined Tesla Autopilot to work on 3D perception.
Oct. 2021	Joined Adobe Emerging Products Group to work on computational photography.
Aug. 2021	Graduated from MIT ( doctoral dissertation ).
Aug. 2021	NeRFactor conditionally accepted to SIGGRAPH Asia.
Dec. 2020	Neural Light Transport (NLT) accepted to TOG, to be presented at SIGGRAPH in August 2021.
Apr. 2019	MoSculp on exhibition in the MIT Museum ( photos ).
Jan. 2019	Awarded Snap Research Fellowship .
Dec. 2018	Oral presentation of GenRe at NeurIPS.
Sep. 2018	MoSculp featured as 9/19 MIT Homepage and covered by Forbes , BBC , etc.
Oct. 2016	Our paper on Bayesian modeling of Alzheimer's disease heterogeneity out in PNAS and covered by Psychology Today , MGH/HMS , etc.
May 2016	Our work on Alzheimer's disease subtypes awarded the Magna Cum Laude Award and oral presentation at ISMRM 2016.

Products

[source]

Full Self-Driving (FSD)
Worked on 3D perception and safety for E2E policies as part of FSD releases to both customer and Robotaxi fleets, contributing to both online models shipped to cars and offline data pipelines scaling to 20K GPUs.
Tesla 2022–2025

[source]

Actually Smart Summon
Helped vehicles perceive static and dynamic 3D scenes, allowing them to autonomously pick up their owners (e.g., from a store exit in the rain) without collisions.
Tesla 2024

[source]

Autopark
Developed the offline system for estimating 3D surroundings and predicting parkable space, enabling autonomous parking into parallel, perpendicular, and angled spots—marked or unmarked—in unseen environments.
Tesla 2023

[source]

[patent] [patent]
High-Fidelity Park Assist
Tackled various aspects of this next-generation, real-time 3D reconstruction system, covering deformable human body, car geometry given only partial views, and large-scale static scenes.
Tesla 2023

Recent Publications

* indicates equal contribution.

[paper] [video] [project] [code] [bibtex]
DiffusionRig: Learning Personalized Priors for Facial Appearance Editing
Zheng Ding, Xuaner Zhang , Zhihao Xia , Lars Jebe , Zhuowen Tu , Xiuming Zhang
CVPR 2023

[paper] [video] [project] [bibtex]
Portrait Reconstruction and Relighting Using the Sun as a Light Stage
Yifan Wang , Aleksander Holynski , Xiuming Zhang , Xuaner Zhang
CVPR 2023

[thesis] [video] [bibtex]
Shape, Reflectance, and Illumination From Appearance
Committee: William T. Freeman , Jonathan T. Barron , Antonio Torralba
MIT Doctoral Dissertation 2021

[paper] [video] [talk] [code] [project] [bibtex]
NeRFactor: Neural Factorization of Shape and Reflectance Under an Unknown Illumination
Xiuming Zhang , Pratul P. Srinivasan , Boyang Deng , Paul Debevec , William T. Freeman , Jonathan T. Barron
TOG 2021 (Proc. SIGGRAPH Asia )

[paper] [video] [code] [demo] [project] [bibtex]
Editing Conditional Radiance Fields
Steven Liu , Xiuming Zhang , Zhoutong Zhang , Richard Zhang , Jun-Yan Zhu , Bryan Russell
ICCV 2021

[paper] [video] [project] [bibtex]
NeRV: Neural Reflectance and Visibility Fields for Relighting and View Synthesis
Pratul P. Srinivasan , Boyang Deng , Xiuming Zhang , Matthew Tancik , Ben Mildenhall , Jonathan T. Barron
CVPR 2021

[paper] [video] [talk] [code] [patent] [project] [bibtex]
Neural Light Transport for Relighting and View Synthesis
Xiuming Zhang , Sean Fanello , Yun-Ta Tsai , Tiancheng Sun , Tianfan Xue , Rohit Pandey , Sergio Orts-Escolano , Philip Davidson , Christoph Rhemann , Paul Debevec , Jonathan T. Barron , Ravi Ramamoorthi , William T. Freeman
TOG 2021 (Presented at SIGGRAPH )

[paper] [video] [project] [bibtex]
Multi-Plane Program Induction With 3D Box Priors
Yikai Li *, Jiayuan Mao *, Xiuming Zhang , William T. Freeman , Joshua B. Tenenbaum , Noah Snavely , Jiajun Wu
NeurIPS 2020

[paper] [video] [talk] [project] [bibtex]
Light Stage Super-Resolution: Continuous High-Frequency Relighting
Tiancheng Sun , Zexiang Xu , Xiuming Zhang , Sean Fanello , Christoph Rhemann , Paul Debevec , Yun-Ta Tsai , Jonathan T. Barron , Ravi Ramamoorthi
TOG 2020 (Proc. SIGGRAPH Asia )

[paper] [supp] [video] [code] [project] [bibtex]
Portrait Shadow Manipulation
Xuaner Zhang , Jonathan T. Barron , Yun-Ta Tsai , Rohit Pandey , Xiuming Zhang , Ren Ng , David E. Jacobs
TOG 2020 (Proc. SIGGRAPH )

[paper] [supp] [code] [project] [bibtex]
Perspective Plane Program Induction From a Single Image
Yikai Li *, Jiayuan Mao *, Xiuming Zhang , William T. Freeman , Joshua B. Tenenbaum , Jiajun Wu
CVPR 2020

[paper] [bibtex]
Latent Atrophy Factors Related to Phenotypical Variants of Posterior Cortical Atrophy
Colin Groot, B. T. Thomas Yeo , Jacob W. Vogel, Xiuming Zhang , Nanbo Sun , Elizabeth C. Mormino , Yolande A. L. Pijnenburg, Bruce L. Miller, Howard J. Rosen, Renaud La Joie, Frederik Barkhof, Philip Scheltens, Wiesje M. van der Flier, Gil D. Rabinovici, Rik Ossenkoppele
Neurology 2020

[paper] [supp] [project] [bibtex]
Program-Guided Image Manipulators
Jiayuan Mao *, Xiuming Zhang *, Yikai Li , William T. Freeman , Joshua B. Tenenbaum , Jiajun Wu
ICCV 2019

[paper] [bibtex]
Reconciling Dimensional and Categorical Models of Autism Heterogeneity: A Brain Connectomics and Behavioral Study
Siyi Tang *, Nanbo Sun *, Dorothea L. Floris, Xiuming Zhang , Adriana Di Martino, B. T. Thomas Yeo
Biological Psychiatry 2019

[paper] [supp] [talk] [code] [project] [bibtex]
Learning to Reconstruct Shapes From Unseen Classes
Xiuming Zhang *, Zhoutong Zhang *, Chengkai Zhang , Joshua B. Tenenbaum , William T. Freeman , Jiajun Wu
NeurIPS 2018
Oral Presentation (Oral/Accepted/Submitted: 30/1011/4856)

[paper] [supp] [thesis] [video] [talk] [demo] [project] [bibtex]
MoSculp: Interactive Visualization of Shape and Time
Xiuming Zhang , Tali Dekel , Tianfan Xue , Andrew Owens , Qiurui He , Jiajun Wu , Stefanie Mueller , William T. Freeman
UIST 2018
Press Coverage: Forbes , BBC , MIT , 9/19 MIT Homepage , etc.
Outreach: MIT Museum

[paper] [code] [project] [bibtex]
Learning Shape Priors for Single-View 3D Completion and Reconstruction
Jiajun Wu *, Chengkai Zhang *, Xiuming Zhang , Zhoutong Zhang , William T. Freeman , Joshua B. Tenenbaum
ECCV 2018

[paper] [code] [project] [bibtex]
Pix3D: Dataset and Methods for Single-Image 3D Shape Modeling
Xingyuan Sun *, Jiajun Wu *, Xiuming Zhang , Zhoutong Zhang , Chengkai Zhang , Tianfan Xue , Joshua B. Tenenbaum , William T. Freeman
CVPR 2018

[paper] [supp] [code] [poster] [bibtex]
Bayesian Model Reveals Latent Atrophy Factors With Dissociable Cognitive Trajectories in Alzheimer's Disease
Xiuming Zhang , Elizabeth C. Mormino , Nanbo Sun , Reisa A. Sperling , Mert R. Sabuncu , B. T. Thomas Yeo
PNAS 2016
Magna Cum Laude Award & Oral Presentation at ISMRM 2016
Press Coverage: Psychology Today , MGH/HMS , etc.

Press Coverage

Side Work

Non-Referred Notes & Side Projects

[code]
Webpage Monitor
A minimal Python webpage monitor that I use to track when a researcher I follow updates their webpage (e.g., adding a new paper).

[code]
This Website
A simple, responsive webpage for academics. Feel free to use it, but on your page, please attribute the source to this page and HTML5 UP .

[doc] [code]
xiuminglib
My Python library for signal/image processing, 3D modeling and rendering with Blender, visualization, etc.

[paper] [code]
Depth-Preserving Style Transfer
An improvement to Johnson et al. [2016] aiming to preserve depth variations during style transfer (MIT 6.869 class project with Ruizhi Liao and Yu Xia ).

[paper]
ReML Estimation in Linear Models
A tutorial on Restricted Maximum Likelihood Estimation in linear regression and Linear Mixed-Effects Models.

[code]
GrabCut
A MATLAB implementation excluding border matting and user editing.

[code]
Basic Photometric Stereo
A MATLAB implementation of the basic photometric stereo algorithm.

[code]
Bandgap of 2D Photonic Crystal
Replicating a classic textbook plot with COMSOL-MATLAB scripting.

Services

Conference Reviewer

EG 2023 , CVPR 2023 , SIGGRAPH Asia 2022 , NeurIPS 2022 , ICML 2022 , ICLR 2022 , ECCV 2022 , CVPR 2022 , SIGGRAPH Asia 2021 , NeurIPS 2021 , MMA 2021 , ICLR 2021 , ICCV 2021 , CVPR 2021 , PG 2020 , NeurIPS 2020 , ECCV 2020 , CVPR 2020 , AAAI 2020 , MM 2019 , ICML 2019 , ICCV 2019 , GLOBECOM 2018

Journal Reviewer

IJCV , TCSVT , TPAMI