Date & Time:
February 28, 2020 10:30 am – 11:30 am
TTIC 526, 6045 S. Kenwood Ave., Chicago, IL,
02/28/2020 10:30 AM 02/28/2020 11:30 AM America/Chicago Xiang Cheng (Berkeley) – Sampling as Optimization and Optimization as Sampling TTIC 526, 6045 S. Kenwood Ave., Chicago, IL,

Sampling as Optimization and Optimization as Sampling

This talk presents a series of results that draw connections between optimization and sampling. In one such result, we show that the Langevin SDE corresponds to the gradient flow of KL divergence with respect to the 2-Wasserstein metric in probability space. This allows us to prove convergence of Langevin MCMC in KL divergence, and even achieve accelerated rates in a similar fashion to Nesterov’s accelerated gradient descent. In the reverse direction, we can also show that Stochastic Gradient Descent may be viewed as the discretization of a certain Stochastic Differential Equation with a state-dependent diffusion matrix that corresponds to the covariance matrix of the sampled stochastic gradient. This theory helps us explain the behavior of SGD in settings such as the training of deep neural networks, where it has been observed that larger noise (in the form of smaller batch-size/larger step-size) gives smaller generalization error.

Based on joint work with Peter Bartlett, Niladri Chatterji, Michael Jordan, Yian Ma, Dong Yin

Host: Nati Srebro

Xiang Cheng

Graduate Student, University of California, Berkeley

I am graduate student in the EECS department at UC Berkeley, co-advised by Peter Bartlett and Michael Jordan. I am interested in the connections between optimization and sampling algorithms for machine learning. Recently, I have been trying use insights from SDE theory to understand the statistical consequences of randomness in algorithms such as Stochastic Gradient Descent.

I am also interested in topics in online learning, bandits, and reinforcement learning and game theory.

Related News & Events

Students posing at competition
UChicago CS News

UChicago Undergrad Team Places Second Overall In Regionals For World’s Largest Programming Competition

Mar 17, 2023
Haifeng Xu
UChicago CS News

New CS and DSI Faculty Haifeng Xu Brings Strategic Intelligence to NeurIPS 2022

Nov 28, 2022
UChicago CS News

UChicago CS Research Finds New Angle on Database Query Processing with Geometry

Nov 08, 2022
UChicago CS News

Asst. Prof. Aloni Cohen Receives Award For Revealing Flaws in Deidentifying Data

Sep 09, 2022
UChicago CS News

UChicago Hosts NSF Workshop on Frontiers of Quantum Advantage

Aug 15, 2022
UChicago CS News

New 2022-23 Faculty Add Expertise in Linguistics, Visualization, Economics, and Data Science Education

Aug 11, 2022
In the News

UChicago Co-Leads $10 Million NSF Institute on Foundations of Data Science

Aug 09, 2022
In the News

Bill Fefferman Comments on New Standards for Quantum-Proof Cryptography

Jul 07, 2022
UChicago CS News

UChicago London Colloquium Features Data Science, Quantum Research

Jul 01, 2022
UChicago CS News

Faculty Bill Fefferman and Chenhao Tan Receive Google Research Scholar Awards

Jun 21, 2022
UChicago CS News

First-Year PhD Student Co-Authors Outstanding Paper Award Winner at TQC 2022

Apr 28, 2022
In the News

Quanta Magazine Features Prof. Bill Fefferman’s Work on Quantum Algorithms

Jan 20, 2022
arrow-down-largearrow-left-largearrow-right-large-greyarrow-right-large-yellowarrow-right-largearrow-right-smallbutton-arrowclosedocumentfacebookfacet-arrow-down-whitefacet-arrow-downPage 1CheckedCheckedicon-apple-t5backgroundLayer 1icon-google-t5icon-office365-t5icon-outlook-t5backgroundLayer 1icon-outlookcom-t5backgroundLayer 1icon-yahoo-t5backgroundLayer 1internal-yellowinternalintranetlinkedinlinkoutpauseplaypresentationsearch-bluesearchshareslider-arrow-nextslider-arrow-prevtwittervideoyoutube