Streaming PCA Review Article

The Proceedings of IEEE posted our review article today on Streaming PCA and Subspace Tracking with Missing Data. It was a great experience to work with Yuejie Chi and Yue Lu on this survey. You can also find a less pretty version on the arxiv.

New paper in Journal of Multivariate Analysis

Congratulations to my student David Hong (and his co-advisor Jeff Fessler) for our published article in the Journal of Multivariate Analysis, titled “Asymptotic performance of PCA for high-dimensional heteroscedastic data.” Heteroscedastic data, where different data points are of differing quality (precisely, have different noise variance), are common in so many interesting big data problems. Sensor network data, medical imaging using historical data, and astronomical imaging are just a few examples. PCA is known to be the maximum likelihood estimate for data with additive Gaussian noise of a single variance across all the data points. This work investigates the performance of PCA when that homoscedastic noise assumption is violated. We give precise predictions for the recovery of subspaces and singular values in a spiked/planted model, and show that vanilla PCA (perhaps unsurprisingly) has suboptimal subspace recovery when the data are heteroscedastic. 

3M Non-Tenured Faculty Award

I am honored to have received 3M’s Non-Tenured Faculty Award, which recognizes outstanding junior faculty nominated by 3M researchers on the basis of their demonstrated record of research, experience, and academic leadership. I look forward to working with 3M Machine Learning researchers to advance data science research!

Publications Update

We have had many exciting publications in the last several months.

My student Dejiao Zhang and I worked with Mario Figueiredo and two other Michigan students on applying OWL regularization in deep networks. The intuition is that since OWL can tie correlated regressors, it should be able to do the same in deep nets that experience a high degree of co-adaptation (and correlation) of nodes in the network. Dejiao presented our paper Learning to Share: Simultaneous Parameter Tying and Sparsification for Deep Learning at ICLR last month and we will present Simultaneous Sparsity and Parameter Tying for Deep Learning using Ordered Weighted L1 Regularization at SSP next month.

With my colleague Johanna Mathieu and her student Greg Ledva, we published a paper in Transactions on Power Systems studying Real-Time Energy Disaggregation of a Distribution Feeder’s Demand Using Online Learning. The work leverages recent results in dynamic online learning where classes of dynamical models are used to apply online learning to the time-varying signal setting. This work can leverage existing sensing structure to improve prediction of distributed energy resources, demand-responsive electric loads and residential solar generation. We also have a book chapter in Energy Markets and Responsive Grids that was written also with my student Zhe Du.

Greg Ongie, David Hong, Dejiao Zhang, and I have been working on adaptive sampling for subspace estimation. If one has a matrix in memory that is large and difficult to access, but you want to compute a low-rank approximation of that matrix, one way is to sketch it by reading only parts of the matrix and computing an approximation. Our paper Enhanced Online Subspace Estimation Via Adaptive Sensing describes an adaptive sampling scheme to do exactly that, and using that scheme along with the GROUSE subspace estimation algorithm, we gave global convergence guarantees to the true underlying low-rank matrix. We will also present Online Estimation of Coherent Subspaces with Adaptive Sampling at SSP next month, which constrains the adaptive samples to be entry-wise and sees similar improvements.

Rounding it out, Zhe Du will be presenting our work with Necmiye Ozay on A Robust Algorithm for Online Switched System Identification at the SYS ID conference in July, and Bob Malinas and David Hong will present our work with Jeff Fessler on Learning Dictionary-Based Unions of Subspaces for Image Denoising at EUSIPCO in September. This spring Amanda Bower presented our work with Lalit Jain on The Landscape of Nonconvex Quadratic Feasibility, studying the minimizers for a non-convex formulation of the preference learning problem; and next week Naveen Murthy presents our work with Greg Ongie and Jeff Fessler on Memory-efficient Splitting Algorithms for Large-Scale Sparsity Regularized Optimization at the CT Meeting. Last fall Greg Ongie, Saket Dewangan, Jeff Fessler and I had a paper Online Dynamic MRI Reconstruction via Robust Subspace Tracking at GlobalSIP, pursuing the interesting idea of online subspace tracking for time-varying signals.

So many exciting research directions that we will continue to pursue!

Congratulations John!

Congratulations to Dr. John Lipor for successfully defending his PhD thesis in September! The title of his work is “Sensing Structured Signals with Active and Ensemble Methods.” In January he will start as Assistant Professor in the Portland State University ECE Department.

Congratulations David!

David Hong was awarded the Richard and Eleanor Towner Prize for Outstanding PhD Research at the Michigan Engineering Graduate Symposium. This is a prize awarded annually across the entire college of engineering to PhD students within about a year of graduation, and the criteria for selection are creativity, innovation, impact on society, and achievement. Congratulations David!

Data-Driven Discovery of Models

Jason Corso and I have been awarded a DARPA D3M grant. Our project is called SPIDER: Subspace Primitives that are Interpretable and DivERse. We will be contributing machine learning software primitives for a system that helps domain experts perform a wide variety of automated data analysis on their datasets. The project has 24 teams and is already off to a great start — We look forward to final system developed by this team over the next few years!

ICML Acceptances, SIAM OPT success

Congratulations to postdoc Greg Ongie, whose excellent work on Variety Models for Matrix Completion has been accepted to ICML. We’re very excited about the potential applications and open problems that we posed in this work. Congratulations also to John Lipor, whose work on Active Subspace Clustering has also been accepted to ICML — I’ve spoken about this work before at the Simons Institute workshop on Interactive Learning. It achieves state of the art clustering error on several benchmark datasets using very few pairwise cluster queries.

We also just finished a week at the SIAM Optimization conference, where our mini-symposium on Non-convex Optimization in Data Analysis was a huge hit. We had a full room for each session and 12 outstanding talks. Thanks to my co-organizers Stephen Wright, Rebecca Willett, and Rob Nowak, and thanks to all the speakers and participants.

MIDAS seminar and new results

Last Friday I gave the MIDAS weekly seminar. You can find the description here, along with the link directly to the recording. I talked about two recent problems I have been working on: First I talked about my work with Ravi Ganti and Rebecca Willett on learning a low-rank matrix that is observed through a monotonic function from partial measurements. This is common in calibration and quantization problems. Follow up work with Nikhil Rao and Rob Nowak in addition generalized this to learning structured single index models. Second, I talked about the work of my student David Hong, co-advised by Jeff Fessler, on the asymptotic performance of PCA with heteroscedastic data. This is common in problems like sensor networks or medical imaging, where different measurements of the same phenomenon are taken with different quality sensing (eg high or low radiation). David has recently posted his paper on arxiv showing predictions of the asymptotic performance; exploiting the structure of these expressions we also showed that asymptotic recovery for a fixed average noise variance is maximized when the noise variances are equal (i.e., when the noise is in fact homoscedastic). Average noise variance is often a practically convenient measure for the overall quality of data, but our results show that it gives an overly optimistic estimate of the performance of PCA for heteroscedastic data.

AAAI and Simons

This week I head to the bay area for two presentations: My collaborator Ravi Ganti will be presenting our work on estimating high dimensional structured single index models.  The week following I’ll be at the Simons workshop on Interactive Learning presenting my work with student John Lipor on active labeling from union of subspace data. I’m looking forward to sharing our work and hearing about all the other interesting work going on in these areas!