Publications

On the Optimization Properties of Optimal-Transport Based Generative Adversarial Linear Networks

P. B.. Supervisor: Guido Montúfar, PhD Thesis, 2024

This thesis is concerned with the analysis of the optimization problem in a class of generative models called Generative Adversarial Networks (GANs). GANs are successful generative models used to learn to mimic a given sampled distribution, and can be linked to Optimal Transport (OT) by the loss they rely on. OT is a reliable means to quantitatively compare two given probability distributions. The overall optimization problem is challenging as it involves a minimax optimization, in a generally highly non- convex-concave framework. In order to alleviate the analysis issues raised by this setting, several means can be employed. This thesis explores some of them in order to study a simplified OT-based generative model. After introducing relevant mathematical tools in Chapter 2, we present in Chapter 3 the general framework of GANs, as well as the different simplifications that can be employed in order to derive analytical guarantees on the optimization system. Two examples, one discrete and one continuous, are investigated in order to illustrate how the analytical framework can be used in very simplified settings. A particularly useful simplification is to reduce the minimax problem to a minimization problem only, by considering the maximization problem as being optimally solved at each time. In OT-based GAN, this typically boils down to analysing a type of distance between probability distributions, the Wasserstein distance. The minimization problem corresponds to finding a distribution that minimizes this distance to a given, target distribution. Chapters 4 and 5 leverage this simplification and study the landscape and convergence properties of a particular, linear OT-based GAN. In this case, the optimization problem can be cast as an overparametrized matrix factorization problem, combining a deep linear neural network with a covariance matrix parametrizations. The landscape of the resulting loss, based on the so-called Bures-Wasserstein distance between the covariance matrices, is investigated in Chapter 4. More precisely, the critical points of the loss are derived, under the appropriate parametrizations. Given a rank constraint on the model covariance matrix, we make explicit the global minimizer to the matrix approximation problem. We find that they are are identical to the minimizers when considering a Euclidean distance between the covariance matrices. The Hessian of the loss at the critical points is derived and numerically investigated, in order to understand what differences the Euclidean and the Bures-Wasserstein distances may introduce in the optimization. In Chapter 5, we analyze the gradient flow and gradient descent optimization schemes of this deep linear neural network under the Bures-Wasserstein loss. Convergence to minimizers of the loss is proven, under additional assumptions on the initialization of the scheme. The rank constraint on the approximation matrix appears here naturally as a bottleneck in the network representing the model covariance matrix. Complementary to the main discussion about GANs, the thesis also includes a chapter on the optimization landscape in a supervised learning setting (Chapter 6 on mode connectivity). Its conclusions may be linked to the GAN discussion in the simplified, limited setting of a minimization-only optimization scheme.

Critical Points and Convergence Analysis of Generative Deep Linear Networks Trained with Bures-Wasserstein-Loss

P. B., Katerina Papagiannouli, Jing An, and Guido Montúfar, ICML, 2023

We consider a deep matrix factorization model of covariance matrices trained with the Bures-Wasserstein distance. While recent works have made advances in the study of the optimization problem for overparametrized low-rank matrix approximation, much emphasis has been placed on discriminative settings and the square loss. In contrast, our model considers another type of loss and connects with the generative setting. We characterize the critical points and minimizers of the Bures-Wasserstein distance over the space of rank-bounded matrices. The Hessian of this loss at low-rank matrices can theoretically blow up, which creates challenges to analyze convergence of gradient optimization methods. We establish convergence results for gradient flow using a smooth perturbative version of the loss as well as convergence results for finite step size gradient descent under certain assumptions on the initial weights.

When are solutions connected in deep networks?

Quyhn N. Nguyen, P. B., and Marco Mondelli, NeurIPS, 2021

The question of how and why the phenomenon of mode connectivity occurs in training deep neural networks has gained remarkable attention in the research community. From a theoretical perspective, two possible explanations have been proposed: (i) the loss function has connected sublevel sets, and (ii) the solutions found by stochastic gradient descent are dropout stable. While these explanations provide insights into the phenomenon, their assumptions are not always satisfied in practice. In particular, the first approach requires the network to have one layer with order of N neurons (N being the number of training samples), while the second one requires the loss to be almost invariant after removing half of the neurons at each layer (up to some rescaling of the remaining ones). In this work, we improve both conditions by exploiting the quality of the features at every intermediate layer together with a milder over-parameterization requirement. More specifically, we show that: (i) under generic assumptions on the features of intermediate layers, it suffices that the last two hidden layers have order of √N neurons, and (ii) if subsets of features at each layer are linearly separable, then almost no over-parameterization is needed to show the connectivity. Our experiments confirm that the proposed condition ensures the connectivity of solutions found by stochastic gradient descent, even in settings where the previous requirements do not hold.

Informative GANs via Structured Regularization of Optimal Transport

P. B., Tao Wu, Thomas Möllenhoff, and Daniel Cremers, Optimal Transport Workshop, NeurIPS, 2019

We tackle the challenge of disentangled representation learning in generative adversarial networks (GANs) from the perspective of regularized optimal transport (OT). Specifically, a smoothed OT loss gives rise to an implicit transportation plan between the latent space and the data space. Based on this theoretical observation, we exploit a structured regularization on the transportation plan to encourage a prescribed latent subspace to be informative. This yields the formulation of a novel informative OT-based GAN. By convex duality, we obtain the equivalent view that this leads to perturbed ground costs favoring sparsity in the informative latent dimensions. Practically, we devise a stable training algorithm for the proposed informative GAN. Our experiments support the hypothesis that such regularizations effectively yield the discovery of disentangled and interpretable latent representations. Our work showcases potential power of a regularized OT framework in the context of generative modeling through its access to the transport plan. Further challenges are addressed in this line.

© Pierre Bréchet. Adapted from Chirpy theme for Jekyll.