We have updated our paper on optimally weighted heteroscedastic PCA and here is the code to run the experiments. In this work we show how to weight data before solving PCA, under a spiked covariance model with heteroscedastic additive noise. Surprisingly, the weights are not inverse noise variance, and neither are they 0/1 discarding the noisiest points, but instead the optimal weights are in between these two standard heuristics.
October 16, 2026 By Laura Balzano
Our paper on Emergent low-rank training dynamics in nonlinear MLPs has been accepted to Neurips 2026. In the paper, we build from our ICML 2024 paper where we proved that deep linear networks have exactly low-rank training dynamics for the whole trajectory of training for problems with a low-rank target, for example a classification problem with few classes compared to the data dimension. In this current paper, we extend those theoretical results to the setting where the MLP has a nonlinear but smooth activation function. While our theory studies two-layer MLPs where we train only the inner weights, we see empirically that this phenomenon is maintained regardless of the depth of the network. We can then train on a compressed model to get similar accuracy results with a tiny fraction of the trainable parameters. Our key insight in both works is that there is a clever initialization that allows one to train compressed deep networks and maintain similar dynamics (or exactly the same dynamics in the linear case) throughout training. Congratulations especially to Alec Xu for all the hard work he put in to this problem. We hope you’ll come see his poster in Sydney!
August 18, 2026 By Laura Balzano
Congratulations Dr. Can Yaras for successfully defending his PhD in July. His dissertation was titled “Low-dimensional Structures of Learning and Computation in Deep Learning,” including work that is already making an impact in the crowded field of deep learning efficiency and algorithm understanding. He will be moving on to a postdoc at Princeton Language and Intelligence.
July 10, 2026 By Laura Balzano
I’m so proud of Dr. Alex Ritchie for his excellent PhD defense! His dissertation is titled “Covariate Structure as Weak Supervision in Estimation Problems,” and includes interesting theoretical and empirical results for Supervised PCA as well as learning nonparametric mixtures. We look forward to all the good Dr. Ritchie will certainly do in the world!
Copyright © 2026 · Streamline Child Theme on Genesis Framework · WordPress · Log in