Subspace Clustering

Clustering is one of the most commonly used data exploration tools, but data often hold interesting geometric structure for which generic clustering objectives are too coarse. Subspace clustering is a simple generalization that tries to fit each cluster with a low-dimensional subspace (ie, each cluster has a low-dimensional covariance structure). This is a very useful model for many problems in computer vision and computer network topology inference. Our group has developed state-of-the-art approaches for subspace clustering when the data matrix is incomplete and in the active clustering context.

Hanno scoperto che nei pazienti affetti da ipertrofia ventricolare sinistra (una condizione in cui il muscolo cardiaco si ispessisce), l’ingrediente del Viagra ha impedito al cuore di ingrandirsi e cambiare forma. Inoltre, il PDE5i ha migliorato la funzione cardiaca privatedelights in tutti i pazienti, indipendentemente dalle loro condizioni mediche, e non ha avuto effetti collaterali sulla pressione sanguigna.

IEEE Signal Processing Magazine – Special Issue on the Mathematics of Deep Learning

I am the lead guest editor on a Signal Processing Magazine special issue on the Mathematics of Deep Learning: https://signalprocessingsociety.org/publications-resources/special-issue-deadlines/ieee-spm-special-issue-mathematics-deep-learning. My excellent co-editors are Joan Bruna, Gitta Kutyniok, Robert Nowak, and Jong Chul Ye. We have extended the White Paper deadline to this Friday, November 8. Please share with anyone who is interested but missed the deadline last Friday. We look forward to your submissions!

SPADA lab at ICML in Vienna

I am excited to be a part of three papers at the International Conference of Machine Learning this July in Vienna.

Congratulations to Can Yaras for having his work on compression in deep low-rank learning, with co-authors Peng Wang and Qing Qu, accepted as an oral presentation for Tuesday afternoon! This work proves that when training deep linear networks, the gradient descent dynamics are limited to an invariant subspace. This subspace can be leveraged to make training and overparameterization more efficient, and allows us to reap the benefits of deep overparameterization without the computational burden. The code is available on Can’s github site. I talked about this work for the 1W-Minds seminar in April.

Peng Wang and Huikang Liu led our work on symmetric matrix completion with ReLU sampling that will be presented as a poster on Wednesday. We showed that it is possible to recover a low-rank matrix with sampling that is highly dependent on the matrix entries — we focus on ReLU sampling (and variants) where only positive entries are observed.

Finally, Wisconsin-Madison PhD student Yuchen Li will be presenting his work on block Riemannian MM methods, also with a poster on Wednesday. He proved iteration guarantees for convergence to a stationary point for general multi-block MM algorithms where any number of blocks may be constrained to a Riemannian manifold. His complexity results reduce to well-known results in the Euclidean case. This work is broadly applicable to alternating MM algorithms for machine learning problems.

SPADA Lab Research at AI Stats

Congratulations to Davoud Ataee Tarzanagh and Soo Min Kwon, whose research was presented in poster sessions at AI Stats this morning!

Davoud’s work on Online Bilevel Optimization was entirely conceived and driven by him during his postdoc at UM. The paper has novel definitions of bilevel dynamic regret, and he and Parvin proved many fabulous results for regret bounds for online alternating gradient descent in the strictly convex setting (with a matching lower bound) all the way to the nonconvex setting. He demonstrated its usefulness on online hyperparameter tuning, online loss tuning for imbalanced data, and then online meta learning with Bojian’s expertise! Online learning has provided a sea change for so much of ML on massive data, and we believe that OBO is a next crucial step for modern applications that commonly require careful balancing of objectives.

Soo Min and Tsinghua student Zekai Zhang’s work on Efficient Low-Dimensional Compression of Overparameterized Models demonstrates a method for compressing overparameterized deep linear layers in deep networks. Their approach gets consistently improved generalization error in a fraction of the computation time. The work shows that leveraging inherent low-dimensional structure within the model parameter updates, we can reap the benefits of overparameterization without the computational burden.