Qing Qu (U of Michigan)- Harnessing Low-Dimensionality for Generalizable and Scientific Generative AI

Abstract: The empirical success of modern generative AI, from diffusion models to Large Language Models (LLMs), often outpaces our classical understanding of how machine learning models generalize from finite, out-of-distribution (OOD) data. This talk introduces a unified mathematical framework identifying intrinsic low-dimensional structures as the primary driver of generalization and a critical lever for advancing scientific AI. First, we deconstruct the generalization mechanism of diffusion models, revealing a training transition from memorization to generalization that effectively breaks the curse of dimensionality. By using a mixture of low-rank Gaussian models, we demonstrate that sample complexity scales linearly with the intrinsic dimension rather than exponentially with the ambient dimension, through establishing a formal equivalence with the canonical subspace clustering problem. Moreover, by exploring nonlinearity in two-layer denoising autoencoders, we uncover how weight structures differ between memorization and generalization. This distinction allows us to provide a unified understanding of how models learn representations and how they generate new data. Second, we characterize the OOD generalization of in-context learning (ICL) in transformers. For linear regression tasks in which vectors lie in low-dimensional subspaces, we show that OOD capabilities emerge from interpolating across training task subspaces. We derive precise conditions under which linear attention models interpolate across distribution shifts, highlighting task diversity as a prerequisite for ICL efficacy. Finally, we translate these theoretical insights into practical guidelines for controlled generation, ensuring model safety and privacy, and solving high-dimensional inverse problems in science and engineering.