Our paper on Emergent low-rank training dynamics in nonlinear MLPs has been accepted to Neurips 2026. In the paper, we build from our ICML 2024 paper where we proved that deep linear networks have exactly low-rank training dynamics for the whole trajectory of training for problems with a low-rank target, for example a classification problem with few classes compared to the data dimension. In this current paper, we extend those theoretical results to the setting where the MLP has a nonlinear but smooth activation function. While our theory studies two-layer MLPs where we train only the inner weights, we see empirically that this phenomenon is maintained regardless of the depth of the network. We can then train on a compressed model to get similar accuracy results with a tiny fraction of the trainable parameters. Our key insight in both works is that there is a clever initialization that allows one to train compressed deep networks and maintain similar dynamics (or exactly the same dynamics in the linear case) throughout training. Congratulations especially to Alec Xu for all the hard work he put in to this problem. We hope you’ll come see his poster in Sydney!
October 16, 2026