Here I list “selected publications” that are representative of my work overall.
Selected Publications
1399621
HJQ26QYG
1
apa
50
date
desc
787
https://web.eecs.umich.edu/~girasole/wp-content/plugins/zotpress/
%7B%22status%22%3A%22success%22%2C%22updateneeded%22%3Afalse%2C%22instance%22%3Afalse%2C%22meta%22%3A%7B%22request_last%22%3A0%2C%22request_next%22%3A0%2C%22used_cache%22%3Atrue%7D%2C%22data%22%3A%5B%7B%22key%22%3A%22W9YJ9EMC%22%2C%22library%22%3A%7B%22id%22%3A1399621%7D%2C%22meta%22%3A%7B%22creatorSummary%22%3A%22Hedayat%20et%20al.%22%2C%22parsedDate%22%3A%222026-07-03%22%2C%22numChildren%22%3A2%7D%2C%22bib%22%3A%22%26lt%3Bdiv%20class%3D%26quot%3Bcsl-bib-body%26quot%3B%20style%3D%26quot%3Bline-height%3A%202%3B%20padding-left%3A%201em%3B%20text-indent%3A-1em%3B%26quot%3B%26gt%3B%5Cn%20%20%26lt%3Bdiv%20class%3D%26quot%3Bcsl-entry%26quot%3B%26gt%3BHedayat%2C%20A.%2C%20Balzano%2C%20L.%2C%20%26amp%3B%20Duraisamy%2C%20K.%20%282026%29.%20%26lt%3Bi%26gt%3BIn-span%20learning%3A%20adapting%20reduced-order%20models%20using%20their%20own%20predictions%26lt%3B%5C%2Fi%26gt%3B%20%28arXiv%3A2607.02937%29.%20arXiv.%20%26lt%3Ba%20class%3D%26%23039%3Bzp-DOIURL%26%23039%3B%20href%3D%26%23039%3Bhttps%3A%5C%2F%5C%2Fdoi.org%5C%2F10.48550%5C%2FarXiv.2607.02937%26%23039%3B%26gt%3Bhttps%3A%5C%2F%5C%2Fdoi.org%5C%2F10.48550%5C%2FarXiv.2607.02937%26lt%3B%5C%2Fa%26gt%3B%26lt%3B%5C%2Fdiv%26gt%3B%5Cn%26lt%3B%5C%2Fdiv%26gt%3B%22%2C%22data%22%3A%7B%22itemType%22%3A%22preprint%22%2C%22title%22%3A%22In-span%20learning%3A%20adapting%20reduced-order%20models%20using%20their%20own%20predictions%22%2C%22creators%22%3A%5B%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Amirpasha%22%2C%22lastName%22%3A%22Hedayat%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Laura%22%2C%22lastName%22%3A%22Balzano%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Karthik%22%2C%22lastName%22%3A%22Duraisamy%22%7D%5D%2C%22abstractNote%22%3A%22Reduced-order%20models%20compress%20high-dimensional%20dynamics%20into%20low-dimensional%20representations%20that%20can%20be%20evaluated%20rapidly%2C%20but%20they%20lose%20accuracy%20when%20online%20dynamics%20drift%20beyond%20the%20training%20data.%20Adaptive%20methods%20address%20this%20by%20updating%20the%20subspace%20online%20with%20external%2C%20out-of-span%20information%2C%20such%20as%20full-order%20corrections%20or%20sensor%20snapshots.%20We%20discovered%20that%20a%20complementary%20and%20previously%20unexploited%20in-span%20adaptation%20channel%20exists%20within%20the%20current%20reduced%20subspace.%20By%20streaming%20the%20model%26%23039%3Bs%20own%20predictions%20through%20an%20incremental%20singular-value%20decomposition%20with%20forgetting%2C%20we%20obtain%20a%20trajectory-informed%20spectral%20preconditioner%2C%20in%20which%20the%20subspace%20is%20unchanged%20but%20the%20basis%20is%20reweighted%20and%20realigned%20toward%20the%20modes%20visited%20by%20the%20dynamics.%20This%20enables%20the%20model%20to%20absorb%20future%20out-of-span%20corrections%20more%20effectively.%20We%20expose%20aspects%20of%20this%20mechanism%20on%20a%20three-dimensional%20spiral%20and%20confirm%20it%20on%20viscous%20Burgers%20and%20Fisher-KPP%20dynamics.%20We%20also%20discuss%20how%20in-span%20learning%20can%20be%20viewed%20as%20a%20dynamical-systems%20analogue%20of%20in-context%20learning.%20More%20broadly%2C%20in-span%20learning%20suggests%20a%20new%20principle%20for%20computational%20science%2C%20revealing%20that%20model-generated%20trajectories%20contain%20more%20usable%20information%20than%20previously%20recognized.%22%2C%22genre%22%3A%22%22%2C%22repository%22%3A%22arXiv%22%2C%22archiveID%22%3A%22arXiv%3A2607.02937%22%2C%22date%22%3A%222026-07-03%22%2C%22DOI%22%3A%2210.48550%5C%2FarXiv.2607.02937%22%2C%22citationKey%22%3A%22%22%2C%22url%22%3A%22http%3A%5C%2F%5C%2Farxiv.org%5C%2Fabs%5C%2F2607.02937%22%2C%22language%22%3A%22%22%2C%22collections%22%3A%5B%22HJQ26QYG%22%2C%22J2WEX3C9%22%5D%2C%22dateModified%22%3A%222026-07-10T21%3A43%3A33Z%22%7D%7D%2C%7B%22key%22%3A%22U666G9AQ%22%2C%22library%22%3A%7B%22id%22%3A1399621%7D%2C%22meta%22%3A%7B%22creatorSummary%22%3A%22Balzano%20et%20al.%22%2C%22parsedDate%22%3A%222026-05%22%2C%22numChildren%22%3A1%7D%2C%22bib%22%3A%22%26lt%3Bdiv%20class%3D%26quot%3Bcsl-bib-body%26quot%3B%20style%3D%26quot%3Bline-height%3A%202%3B%20padding-left%3A%201em%3B%20text-indent%3A-1em%3B%26quot%3B%26gt%3B%5Cn%20%20%26lt%3Bdiv%20class%3D%26quot%3Bcsl-entry%26quot%3B%26gt%3BBalzano%2C%20L.%2C%20Ding%2C%20T.%2C%20Haeffele%2C%20B.%20D.%2C%20Kwon%2C%20S.%20M.%2C%20Qu%2C%20Q.%2C%20Wang%2C%20P.%2C%20Wang%2C%20Z.%2C%20%26amp%3B%20Yaras%2C%20C.%20%282026%29.%20An%20Overview%20of%20Low-Rank%20Structures%20in%20the%20Training%20and%20Adaptation%20of%20Large%20Models%3A%20From%20Implicit%20Low-Dimensionality%20to%20Efficient%20Training%20and%20Fine-Tuning%20%5BSpecial%20Issue%20on%20the%20Mathematics%20of%20Deep%20Learning%5D.%20%26lt%3Bi%26gt%3BIEEE%20Signal%20Processing%20Magazine%26lt%3B%5C%2Fi%26gt%3B%2C%20%26lt%3Bi%26gt%3B43%26lt%3B%5C%2Fi%26gt%3B%283%29%2C%2052%26%23x2013%3B70.%20%26lt%3Ba%20class%3D%26%23039%3Bzp-DOIURL%26%23039%3B%20href%3D%26%23039%3Bhttps%3A%5C%2F%5C%2Fdoi.org%5C%2F10.1109%5C%2FMSP.2026.3666749%26%23039%3B%26gt%3Bhttps%3A%5C%2F%5C%2Fdoi.org%5C%2F10.1109%5C%2FMSP.2026.3666749%26lt%3B%5C%2Fa%26gt%3B%26lt%3B%5C%2Fdiv%26gt%3B%5Cn%26lt%3B%5C%2Fdiv%26gt%3B%22%2C%22data%22%3A%7B%22itemType%22%3A%22journalArticle%22%2C%22title%22%3A%22An%20Overview%20of%20Low-Rank%20Structures%20in%20the%20Training%20and%20Adaptation%20of%20Large%20Models%3A%20From%20Implicit%20Low-Dimensionality%20to%20Efficient%20Training%20and%20Fine-Tuning%20%5BSpecial%20Issue%20on%20the%20Mathematics%20of%20Deep%20Learning%5D%22%2C%22creators%22%3A%5B%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Laura%22%2C%22lastName%22%3A%22Balzano%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Tianjiao%22%2C%22lastName%22%3A%22Ding%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Benjamin%20D.%22%2C%22lastName%22%3A%22Haeffele%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Soo%20Min%22%2C%22lastName%22%3A%22Kwon%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Qing%22%2C%22lastName%22%3A%22Qu%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Peng%22%2C%22lastName%22%3A%22Wang%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Zhangyang%22%2C%22lastName%22%3A%22Wang%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Can%22%2C%22lastName%22%3A%22Yaras%22%7D%5D%2C%22abstractNote%22%3A%22The%20advent%20of%20deep%20learning%20has%20immeasurably%20changed%20the%20ways%20we%20process%20data%20in%20signal%20processing%20and%20machine%20learning.%20However%2C%20training%20and%20deploying%20modern%20deep%20learning%20models%20demand%20substantial%20computational%20resources%2C%20raising%20concerns%20about%20exorbitant%20training%20costs%2C%20GPU%20shortages%2C%20and%20heightened%20energy%20consumption.%20Several%20lines%20of%20research%20over%20the%20last%20decade%20have%20explored%20the%20emergence%20of%20low-dimensional%20structures%20during%20the%20training%20process%2C%20where%20basic%20elements%20such%20as%20weight%20matrices%20and%20representations%20tend%20to%20be%20approximately%20low%20rank%20even%20though%20not%20explicitly%20trained%20to%20be.%20These%20low-dimensional%20structures%20arise%20in%20part%20due%20to%20the%20implicit%20bias%20of%20the%20methods%20used%20to%20train%20deep%20networks%2C%20providing%20the%20potential%20to%20partially%20explain%20why%20deep%20models%20need%20fewer%20samples%20than%20the%20number%20of%20model%20parameters.%20This%20implicit%20low%20dimensionality%20has%20inspired%20the%20exploration%20of%20low-rank%20structures%20in%20training%20and%20fine-tuning%20large-scale%20deep%20learning%20models%20more%20efficiently.%20In%20this%20article%2C%20we%20review%20recent%20exciting%20advances%20in%20using%20low-rank%20structure%20in%20deep%20learning%20and%20aim%20to%20clarify%20the%20mathematical%20foundations%20underlying%20their%20design.%20Specifically%2C%20we%20highlight%20key%20insights%20from%20a%20rich%20line%20of%20research%20focused%20on%20theoretically%20understanding%20and%20leveraging%20low-rank%20structures%20in%20deep%20learning%20both%20at%20every%20iteration%20of%20training%20as%20well%20as%20at%20the%20global%20minimum.%22%2C%22date%22%3A%222026-05%22%2C%22section%22%3A%22%22%2C%22partNumber%22%3A%22%22%2C%22partTitle%22%3A%22%22%2C%22DOI%22%3A%2210.1109%5C%2FMSP.2026.3666749%22%2C%22citationKey%22%3A%22%22%2C%22url%22%3A%22https%3A%5C%2F%5C%2Fieeexplore.ieee.org%5C%2Fdocument%5C%2F11561030%22%2C%22PMID%22%3A%22%22%2C%22PMCID%22%3A%22%22%2C%22ISSN%22%3A%221558-0792%22%2C%22language%22%3A%22%22%2C%22collections%22%3A%5B%22DZFDBB6V%22%2C%22HJQ26QYG%22%5D%2C%22dateModified%22%3A%222026-06-15T23%3A07%3A24Z%22%7D%7D%2C%7B%22key%22%3A%22FNAUTBRG%22%2C%22library%22%3A%7B%22id%22%3A1399621%7D%2C%22meta%22%3A%7B%22creatorSummary%22%3A%22Yaras%20et%20al.%22%2C%22parsedDate%22%3A%222025-12-02%22%2C%22numChildren%22%3A1%7D%2C%22bib%22%3A%22%26lt%3Bdiv%20class%3D%26quot%3Bcsl-bib-body%26quot%3B%20style%3D%26quot%3Bline-height%3A%202%3B%20padding-left%3A%201em%3B%20text-indent%3A-1em%3B%26quot%3B%26gt%3B%5Cn%20%20%26lt%3Bdiv%20class%3D%26quot%3Bcsl-entry%26quot%3B%26gt%3BYaras%2C%20C.%2C%20Xu%2C%20A.%2C%20Abillama%2C%20P.%2C%20Lee%2C%20C.%2C%20%26amp%3B%20Balzano%2C%20L.%20%282025%29.%20MonarchAttention%3A%20Zero-Shot%20Conversion%20to%20Fast%2C%20Hardware-Aware%20Structured%20Attention.%20%26lt%3Bi%26gt%3BAdvances%20in%20Neural%20Information%20Processing%20Systems%26lt%3B%5C%2Fi%26gt%3B%2C%20%26lt%3Bi%26gt%3B38%26lt%3B%5C%2Fi%26gt%3B%2C%203445%26%23x2013%3B3471.%20%26lt%3Ba%20class%3D%26%23039%3Bzp-ItemURL%26%23039%3B%20href%3D%26%23039%3Bhttps%3A%5C%2F%5C%2Fpapers.nips.cc%5C%2Fpaper_files%5C%2Fpaper%5C%2F2025%5C%2Fhash%5C%2F050f8591be3874b52fdac4e1060eeb29-Abstract-Conference.html%26%23039%3B%26gt%3Bhttps%3A%5C%2F%5C%2Fpapers.nips.cc%5C%2Fpaper_files%5C%2Fpaper%5C%2F2025%5C%2Fhash%5C%2F050f8591be3874b52fdac4e1060eeb29-Abstract-Conference.html%26lt%3B%5C%2Fa%26gt%3B%26lt%3B%5C%2Fdiv%26gt%3B%5Cn%26lt%3B%5C%2Fdiv%26gt%3B%22%2C%22data%22%3A%7B%22itemType%22%3A%22journalArticle%22%2C%22title%22%3A%22MonarchAttention%3A%20Zero-Shot%20Conversion%20to%20Fast%2C%20Hardware-Aware%20Structured%20Attention%22%2C%22creators%22%3A%5B%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Can%22%2C%22lastName%22%3A%22Yaras%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Alec%22%2C%22lastName%22%3A%22Xu%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Pierre%22%2C%22lastName%22%3A%22Abillama%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Changwoo%22%2C%22lastName%22%3A%22Lee%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Laura%22%2C%22lastName%22%3A%22Balzano%22%7D%5D%2C%22abstractNote%22%3A%22%22%2C%22date%22%3A%222025-12-02%22%2C%22section%22%3A%22%22%2C%22partNumber%22%3A%22%22%2C%22partTitle%22%3A%22%22%2C%22DOI%22%3A%22%22%2C%22citationKey%22%3A%22%22%2C%22url%22%3A%22https%3A%5C%2F%5C%2Fpapers.nips.cc%5C%2Fpaper_files%5C%2Fpaper%5C%2F2025%5C%2Fhash%5C%2F050f8591be3874b52fdac4e1060eeb29-Abstract-Conference.html%22%2C%22PMID%22%3A%22%22%2C%22PMCID%22%3A%22%22%2C%22ISSN%22%3A%22%22%2C%22language%22%3A%22en%22%2C%22collections%22%3A%5B%22DZFDBB6V%22%2C%22HJQ26QYG%22%5D%2C%22dateModified%22%3A%222026-05-08T14%3A51%3A17Z%22%7D%7D%2C%7B%22key%22%3A%22HRSVSCMM%22%2C%22library%22%3A%7B%22id%22%3A1399621%7D%2C%22meta%22%3A%7B%22creatorSummary%22%3A%22Gilman%20et%20al.%22%2C%22parsedDate%22%3A%222025-06-30%22%2C%22numChildren%22%3A1%7D%2C%22bib%22%3A%22%26lt%3Bdiv%20class%3D%26quot%3Bcsl-bib-body%26quot%3B%20style%3D%26quot%3Bline-height%3A%202%3B%20padding-left%3A%201em%3B%20text-indent%3A-1em%3B%26quot%3B%26gt%3B%5Cn%20%20%26lt%3Bdiv%20class%3D%26quot%3Bcsl-entry%26quot%3B%26gt%3BGilman%2C%20K.%2C%20Burer%2C%20S.%2C%20%26amp%3B%20Balzano%2C%20L.%20%282025%29.%20A%20Semidefinite%20Relaxation%20for%20Sums%20of%20Heterogeneous%20Quadratic%20Forms%20on%20the%20Stiefel%20Manifold.%20%26lt%3Bi%26gt%3BSIAM%20Journal%20on%20Matrix%20Analysis%20and%20Applications%26lt%3B%5C%2Fi%26gt%3B%2C%201091%26%23x2013%3B1116.%20%26lt%3Ba%20class%3D%26%23039%3Bzp-DOIURL%26%23039%3B%20href%3D%26%23039%3Bhttps%3A%5C%2F%5C%2Fdoi.org%5C%2F10.1137%5C%2F23M1545136%26%23039%3B%26gt%3Bhttps%3A%5C%2F%5C%2Fdoi.org%5C%2F10.1137%5C%2F23M1545136%26lt%3B%5C%2Fa%26gt%3B%26lt%3B%5C%2Fdiv%26gt%3B%5Cn%26lt%3B%5C%2Fdiv%26gt%3B%22%2C%22data%22%3A%7B%22itemType%22%3A%22journalArticle%22%2C%22title%22%3A%22A%20Semidefinite%20Relaxation%20for%20Sums%20of%20Heterogeneous%20Quadratic%20Forms%20on%20the%20Stiefel%20Manifold%22%2C%22creators%22%3A%5B%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Kyle%22%2C%22lastName%22%3A%22Gilman%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Samuel%22%2C%22lastName%22%3A%22Burer%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Laura%22%2C%22lastName%22%3A%22Balzano%22%7D%5D%2C%22abstractNote%22%3A%22%22%2C%22date%22%3A%222025-06-30%22%2C%22section%22%3A%22%22%2C%22partNumber%22%3A%22%22%2C%22partTitle%22%3A%22%22%2C%22DOI%22%3A%2210.1137%5C%2F23M1545136%22%2C%22citationKey%22%3A%22%22%2C%22url%22%3A%22https%3A%5C%2F%5C%2Fepubs.siam.org%5C%2Fdoi%5C%2F10.1137%5C%2F23M1545136%22%2C%22PMID%22%3A%22%22%2C%22PMCID%22%3A%22%22%2C%22ISSN%22%3A%220895-4798%22%2C%22language%22%3A%22%22%2C%22collections%22%3A%5B%22DZFDBB6V%22%2C%22HJQ26QYG%22%5D%2C%22dateModified%22%3A%222025-04-17T14%3A14%3A49Z%22%7D%7D%2C%7B%22key%22%3A%224DADRUF5%22%2C%22library%22%3A%7B%22id%22%3A1399621%7D%2C%22meta%22%3A%7B%22creatorSummary%22%3A%22Yaras%20et%20al.%22%2C%22parsedDate%22%3A%222024-06-06%22%2C%22numChildren%22%3A1%7D%2C%22bib%22%3A%22%26lt%3Bdiv%20class%3D%26quot%3Bcsl-bib-body%26quot%3B%20style%3D%26quot%3Bline-height%3A%202%3B%20padding-left%3A%201em%3B%20text-indent%3A-1em%3B%26quot%3B%26gt%3B%5Cn%20%20%26lt%3Bdiv%20class%3D%26quot%3Bcsl-entry%26quot%3B%26gt%3BYaras%2C%20C.%2C%20Wang%2C%20P.%2C%20Balzano%2C%20L.%2C%20%26amp%3B%20Qu%2C%20Q.%20%282024%2C%20June%206%29.%20%26lt%3Bi%26gt%3BCompressible%20Dynamics%20in%20Deep%20Overparameterized%20Low-Rank%20Learning%20%26amp%3B%20Adaptation%26lt%3B%5C%2Fi%26gt%3B.%20Forty-first%20International%20Conference%20on%20Machine%20Learning.%20%26lt%3Ba%20class%3D%26%23039%3Bzp-ItemURL%26%23039%3B%20href%3D%26%23039%3Bhttps%3A%5C%2F%5C%2Fopenreview.net%5C%2Fforum%3Fid%3DuDkXoZMzBv%26%23039%3B%26gt%3Bhttps%3A%5C%2F%5C%2Fopenreview.net%5C%2Fforum%3Fid%3DuDkXoZMzBv%26lt%3B%5C%2Fa%26gt%3B%26lt%3B%5C%2Fdiv%26gt%3B%5Cn%26lt%3B%5C%2Fdiv%26gt%3B%22%2C%22data%22%3A%7B%22itemType%22%3A%22conferencePaper%22%2C%22title%22%3A%22Compressible%20Dynamics%20in%20Deep%20Overparameterized%20Low-Rank%20Learning%20%26%20Adaptation%22%2C%22creators%22%3A%5B%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Can%22%2C%22lastName%22%3A%22Yaras%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Peng%22%2C%22lastName%22%3A%22Wang%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Laura%22%2C%22lastName%22%3A%22Balzano%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Qing%22%2C%22lastName%22%3A%22Qu%22%7D%5D%2C%22abstractNote%22%3A%22While%20overparameterization%20in%20machine%20learning%20models%20offers%20great%20benefits%20in%20terms%20of%20optimization%20and%20generalization%2C%20it%20also%20leads%20to%20increased%20computational%20requirements%20as%20model%20sizes%20grow.%20In%20this%20work%2C%20we%20show%20that%20by%20leveraging%20the%20inherent%20low-dimensional%20structures%20of%20data%20and%20compressible%20dynamics%20within%20the%20model%20parameters%2C%20we%20can%20reap%20the%20benefits%20of%20overparameterization%20without%20the%20computational%20burdens.%20In%20practice%2C%20we%20demonstrate%20the%20effectiveness%20of%20this%20approach%20for%20deep%20low-rank%20matrix%20completion%20as%20well%20as%20fine-tuning%20language%20models.%20Our%20approach%20is%20grounded%20in%20theoretical%20findings%20for%20deep%20overparameterized%20low-rank%20matrix%20recovery%2C%20where%20we%20show%20that%20the%20learning%20dynamics%20of%20each%20weight%20matrix%20are%20confined%20to%20an%20invariant%20low-dimensional%20subspace.%20Consequently%2C%20we%20can%20construct%20and%20train%20compact%2C%20highly%20compressed%20factorizations%20possessing%20the%20same%20benefits%20as%20their%20overparameterized%20counterparts.%20In%20the%20context%20of%20deep%20matrix%20completion%2C%20our%20technique%20substantially%20improves%20training%20efficiency%20while%20retaining%20the%20advantages%20of%20overparameterization.%20For%20language%20model%20fine-tuning%2C%20we%20propose%20a%20method%20called%20%26quot%3BDeep%20LoRA%26quot%3B%2C%20which%20improves%20the%20existing%20low-rank%20adaptation%20%28LoRA%29%20technique%2C%20leading%20to%20reduced%20overfitting%20and%20a%20simplified%20hyperparameter%20setup%2C%20while%20maintaining%20comparable%20efficiency.%20We%20validate%20the%20effectiveness%20of%20Deep%20LoRA%20on%20natural%20language%20tasks%2C%20particularly%20when%20fine-tuning%20with%20limited%20data.%22%2C%22proceedingsTitle%22%3A%22%22%2C%22conferenceName%22%3A%22Forty-first%20International%20Conference%20on%20Machine%20Learning%22%2C%22date%22%3A%222024%5C%2F06%5C%2F06%22%2C%22eventPlace%22%3A%22%22%2C%22DOI%22%3A%22%22%2C%22ISBN%22%3A%22%22%2C%22citationKey%22%3A%22%22%2C%22url%22%3A%22https%3A%5C%2F%5C%2Fopenreview.net%5C%2Fforum%3Fid%3DuDkXoZMzBv%22%2C%22ISSN%22%3A%22%22%2C%22language%22%3A%22en%22%2C%22collections%22%3A%5B%22DZFDBB6V%22%2C%22HJQ26QYG%22%5D%2C%22dateModified%22%3A%222024-06-25T14%3A12%3A40Z%22%7D%7D%2C%7B%22key%22%3A%22QWEB5FTR%22%2C%22library%22%3A%7B%22id%22%3A1399621%7D%2C%22meta%22%3A%7B%22creatorSummary%22%3A%22Liu%20et%20al.%22%2C%22parsedDate%22%3A%222024-06-06%22%2C%22numChildren%22%3A1%7D%2C%22bib%22%3A%22%26lt%3Bdiv%20class%3D%26quot%3Bcsl-bib-body%26quot%3B%20style%3D%26quot%3Bline-height%3A%202%3B%20padding-left%3A%201em%3B%20text-indent%3A-1em%3B%26quot%3B%26gt%3B%5Cn%20%20%26lt%3Bdiv%20class%3D%26quot%3Bcsl-entry%26quot%3B%26gt%3BLiu%2C%20H.%2C%20Wang%2C%20P.%2C%20Huang%2C%20L.%2C%20Qu%2C%20Q.%2C%20%26amp%3B%20Balzano%2C%20L.%20%282024%2C%20June%206%29.%20%26lt%3Bi%26gt%3BSymmetric%20Matrix%20Completion%20with%20ReLU%20Sampling%26lt%3B%5C%2Fi%26gt%3B.%20Forty-first%20International%20Conference%20on%20Machine%20Learning.%20%26lt%3Ba%20class%3D%26%23039%3Bzp-ItemURL%26%23039%3B%20href%3D%26%23039%3Bhttps%3A%5C%2F%5C%2Fopenreview.net%5C%2Fforum%3Fid%3DVxI0gInNlh%26%23039%3B%26gt%3Bhttps%3A%5C%2F%5C%2Fopenreview.net%5C%2Fforum%3Fid%3DVxI0gInNlh%26lt%3B%5C%2Fa%26gt%3B%26lt%3B%5C%2Fdiv%26gt%3B%5Cn%26lt%3B%5C%2Fdiv%26gt%3B%22%2C%22data%22%3A%7B%22itemType%22%3A%22conferencePaper%22%2C%22title%22%3A%22Symmetric%20Matrix%20Completion%20with%20ReLU%20Sampling%22%2C%22creators%22%3A%5B%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Huikang%22%2C%22lastName%22%3A%22Liu%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Peng%22%2C%22lastName%22%3A%22Wang%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Longxiu%22%2C%22lastName%22%3A%22Huang%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Qing%22%2C%22lastName%22%3A%22Qu%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Laura%22%2C%22lastName%22%3A%22Balzano%22%7D%5D%2C%22abstractNote%22%3A%22We%20study%20the%20problem%20of%20symmetric%20positive%20semi-definite%20low-rank%20matrix%20completion%20%28MC%29%20with%20deterministic%20entry-dependent%20sampling.%20In%20particular%2C%20we%20consider%20rectified%20linear%20unit%20%28ReLU%29%20sampling%2C%20where%20only%20positive%20entries%20are%20observed%2C%20as%20well%20as%20a%20generalization%20to%20threshold-based%20sampling.%20We%20first%20empirically%20demonstrate%20that%20the%20landscape%20of%20this%20MC%20problem%20is%20not%20globally%20benign%3A%20Gradient%20descent%20%28GD%29%20with%20random%20initialization%20will%20generally%20converge%20to%20stationary%20points%20that%20are%20not%20globally%20optimal.%20Nevertheless%2C%20we%20prove%20that%20when%20the%20matrix%20factor%20with%20a%20small%20rank%20satisfies%20mild%20assumptions%2C%20the%20nonconvex%20objective%20function%20is%20geodesically%20strongly%20convex%20on%20the%20quotient%20manifold%20in%20a%20neighborhood%20of%20a%20planted%20low-rank%20matrix.%20Moreover%2C%20we%20show%20that%20our%20assumptions%20are%20satisfied%20by%20a%20matrix%20factor%20with%20i.i.d.%20Gaussian%20entries.%20Finally%2C%20we%20develop%20a%20tailor-designed%20initialization%20for%20GD%20to%20solve%20our%20studied%20formulation%2C%20which%20empirically%20always%20achieves%20convergence%20to%20the%20global%20minima.%20We%20also%20conduct%20extensive%20experiments%20and%20compare%20MC%20methods%2C%20investigating%20convergence%20and%20completion%20performance%20with%20respect%20to%20initialization%2C%20noise%20level%2C%20dimension%2C%20and%20rank.%22%2C%22proceedingsTitle%22%3A%22%22%2C%22conferenceName%22%3A%22Forty-first%20International%20Conference%20on%20Machine%20Learning%22%2C%22date%22%3A%222024%5C%2F06%5C%2F06%22%2C%22eventPlace%22%3A%22%22%2C%22DOI%22%3A%22%22%2C%22ISBN%22%3A%22%22%2C%22citationKey%22%3A%22%22%2C%22url%22%3A%22https%3A%5C%2F%5C%2Fopenreview.net%5C%2Fforum%3Fid%3DVxI0gInNlh%22%2C%22ISSN%22%3A%22%22%2C%22language%22%3A%22en%22%2C%22collections%22%3A%5B%22DZFDBB6V%22%2C%22HJQ26QYG%22%5D%2C%22dateModified%22%3A%222024-06-25T14%3A12%3A02Z%22%7D%7D%2C%7B%22key%22%3A%22GCNVEBMA%22%2C%22library%22%3A%7B%22id%22%3A1399621%7D%2C%22meta%22%3A%7B%22creatorSummary%22%3A%22Kwon%20et%20al.%22%2C%22parsedDate%22%3A%222024-04-18%22%2C%22numChildren%22%3A1%7D%2C%22bib%22%3A%22%26lt%3Bdiv%20class%3D%26quot%3Bcsl-bib-body%26quot%3B%20style%3D%26quot%3Bline-height%3A%202%3B%20padding-left%3A%201em%3B%20text-indent%3A-1em%3B%26quot%3B%26gt%3B%5Cn%20%20%26lt%3Bdiv%20class%3D%26quot%3Bcsl-entry%26quot%3B%26gt%3BKwon%2C%20S.%20M.%2C%20Zhang%2C%20Z.%2C%20Song%2C%20D.%2C%20Balzano%2C%20L.%2C%20%26amp%3B%20Qu%2C%20Q.%20%282024%29.%20Efficient%20Low-Dimensional%20Compression%20of%20Overparameterized%20Models.%20%26lt%3Bi%26gt%3BProceedings%20of%20The%2027th%20International%20Conference%20on%20Artificial%20Intelligence%20and%20Statistics%26lt%3B%5C%2Fi%26gt%3B%2C%201009%26%23x2013%3B1017.%20%26lt%3Ba%20class%3D%26%23039%3Bzp-ItemURL%26%23039%3B%20href%3D%26%23039%3Bhttps%3A%5C%2F%5C%2Fproceedings.mlr.press%5C%2Fv238%5C%2Fmin-kwon24a.html%26%23039%3B%26gt%3Bhttps%3A%5C%2F%5C%2Fproceedings.mlr.press%5C%2Fv238%5C%2Fmin-kwon24a.html%26lt%3B%5C%2Fa%26gt%3B%26lt%3B%5C%2Fdiv%26gt%3B%5Cn%26lt%3B%5C%2Fdiv%26gt%3B%22%2C%22data%22%3A%7B%22itemType%22%3A%22conferencePaper%22%2C%22title%22%3A%22Efficient%20Low-Dimensional%20Compression%20of%20Overparameterized%20Models%22%2C%22creators%22%3A%5B%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Soo%20Min%22%2C%22lastName%22%3A%22Kwon%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Zekai%22%2C%22lastName%22%3A%22Zhang%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Dogyoon%22%2C%22lastName%22%3A%22Song%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Laura%22%2C%22lastName%22%3A%22Balzano%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Qing%22%2C%22lastName%22%3A%22Qu%22%7D%5D%2C%22abstractNote%22%3A%22In%20this%20work%2C%20we%20present%20a%20novel%20approach%20for%20compressing%20overparameterized%20models%2C%20developed%20through%20studying%20their%20learning%20dynamics.%20We%20observe%20that%20for%20many%20deep%20models%2C%20updates%20to%20the%20weight%20matrices%20occur%20within%20a%20low-dimensional%20invariant%20subspace.%20For%20deep%20linear%20models%2C%20we%20demonstrate%20that%20their%20principal%20components%20are%20fitted%20incrementally%20within%20a%20small%20subspace%2C%20and%20use%20these%20insights%20to%20propose%20a%20compression%20algorithm%20for%20deep%20linear%20networks%20that%20involve%20decreasing%20the%20width%20of%20their%20intermediate%20layers.%20We%20empirically%20evaluate%20the%20effectiveness%20of%20our%20compression%20technique%20on%20matrix%20recovery%20problems.%20Remarkably%2C%20by%20using%20an%20initialization%20that%20exploits%20the%20structure%20of%20the%20problem%2C%20we%20observe%20that%20our%20compressed%20network%20converges%20faster%20than%20the%20original%20network%2C%20consistently%20yielding%20smaller%20recovery%20errors.%20We%20substantiate%20this%20observation%20by%20developing%20a%20theory%20focused%20on%20deep%20matrix%20factorization.%20Finally%2C%20we%20empirically%20demonstrate%20how%20our%20compressed%20model%20has%20the%20potential%20to%20improve%20the%20utility%20of%20deep%20nonlinear%20models.%20Overall%2C%20our%20algorithm%20improves%20the%20training%20efficiency%20by%20more%20than%202x%2C%20without%20compromising%20generalization.%22%2C%22proceedingsTitle%22%3A%22Proceedings%20of%20The%2027th%20International%20Conference%20on%20Artificial%20Intelligence%20and%20Statistics%22%2C%22conferenceName%22%3A%22International%20Conference%20on%20Artificial%20Intelligence%20and%20Statistics%22%2C%22date%22%3A%222024-04-18%22%2C%22eventPlace%22%3A%22%22%2C%22DOI%22%3A%22%22%2C%22ISBN%22%3A%22%22%2C%22citationKey%22%3A%22%22%2C%22url%22%3A%22https%3A%5C%2F%5C%2Fproceedings.mlr.press%5C%2Fv238%5C%2Fmin-kwon24a.html%22%2C%22ISSN%22%3A%222640-3498%22%2C%22language%22%3A%22en%22%2C%22collections%22%3A%5B%22DZFDBB6V%22%2C%22HJQ26QYG%22%5D%2C%22dateModified%22%3A%222024-04-26T17%3A45%3A38Z%22%7D%7D%2C%7B%22key%22%3A%225NMCPZ4B%22%2C%22library%22%3A%7B%22id%22%3A1399621%7D%2C%22meta%22%3A%7B%22creatorSummary%22%3A%22Hong%20et%20al.%22%2C%22parsedDate%22%3A%222023-03-31%22%2C%22numChildren%22%3A1%7D%2C%22bib%22%3A%22%26lt%3Bdiv%20class%3D%26quot%3Bcsl-bib-body%26quot%3B%20style%3D%26quot%3Bline-height%3A%202%3B%20padding-left%3A%201em%3B%20text-indent%3A-1em%3B%26quot%3B%26gt%3B%5Cn%20%20%26lt%3Bdiv%20class%3D%26quot%3Bcsl-entry%26quot%3B%26gt%3BHong%2C%20D.%2C%20Yang%2C%20F.%2C%20Fessler%2C%20J.%20A.%2C%20%26amp%3B%20Balzano%2C%20L.%20%282023%29.%20Optimally%20Weighted%20PCA%20for%20High-Dimensional%20Heteroscedastic%20Data.%20%26lt%3Bi%26gt%3BSIAM%20Journal%20on%20Mathematics%20of%20Data%20Science%26lt%3B%5C%2Fi%26gt%3B%2C%20%26lt%3Bi%26gt%3B5%26lt%3B%5C%2Fi%26gt%3B%281%29%2C%20222%26%23x2013%3B250.%20%26lt%3Ba%20class%3D%26%23039%3Bzp-DOIURL%26%23039%3B%20href%3D%26%23039%3Bhttps%3A%5C%2F%5C%2Fdoi.org%5C%2F10.1137%5C%2F22M1470244%26%23039%3B%26gt%3Bhttps%3A%5C%2F%5C%2Fdoi.org%5C%2F10.1137%5C%2F22M1470244%26lt%3B%5C%2Fa%26gt%3B%26lt%3B%5C%2Fdiv%26gt%3B%5Cn%26lt%3B%5C%2Fdiv%26gt%3B%22%2C%22data%22%3A%7B%22itemType%22%3A%22journalArticle%22%2C%22title%22%3A%22Optimally%20Weighted%20PCA%20for%20High-Dimensional%20Heteroscedastic%20Data%22%2C%22creators%22%3A%5B%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22David%22%2C%22lastName%22%3A%22Hong%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Fan%22%2C%22lastName%22%3A%22Yang%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Jeffrey%20A.%22%2C%22lastName%22%3A%22Fessler%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Laura%22%2C%22lastName%22%3A%22Balzano%22%7D%5D%2C%22abstractNote%22%3A%22%22%2C%22date%22%3A%222023-03-31%22%2C%22section%22%3A%22%22%2C%22partNumber%22%3A%22%22%2C%22partTitle%22%3A%22%22%2C%22DOI%22%3A%2210.1137%5C%2F22M1470244%22%2C%22citationKey%22%3A%22%22%2C%22url%22%3A%22https%3A%5C%2F%5C%2Fepubs-siam-org.proxy.lib.umich.edu%5C%2Fdoi%5C%2Fabs%5C%2F10.1137%5C%2F22M1470244%22%2C%22PMID%22%3A%22%22%2C%22PMCID%22%3A%22%22%2C%22ISSN%22%3A%22%22%2C%22language%22%3A%22%22%2C%22collections%22%3A%5B%22427SEM27%22%2C%22DZFDBB6V%22%2C%22HJQ26QYG%22%5D%2C%22dateModified%22%3A%222024-03-08T19%3A58%3A40Z%22%7D%7D%2C%7B%22key%22%3A%22HXS2JBFQ%22%2C%22library%22%3A%7B%22id%22%3A1399621%7D%2C%22meta%22%3A%7B%22creatorSummary%22%3A%22Wang%20et%20al.%22%2C%22parsedDate%22%3A%222022-06-28%22%2C%22numChildren%22%3A1%7D%2C%22bib%22%3A%22%26lt%3Bdiv%20class%3D%26quot%3Bcsl-bib-body%26quot%3B%20style%3D%26quot%3Bline-height%3A%202%3B%20padding-left%3A%201em%3B%20text-indent%3A-1em%3B%26quot%3B%26gt%3B%5Cn%20%20%26lt%3Bdiv%20class%3D%26quot%3Bcsl-entry%26quot%3B%26gt%3BWang%2C%20P.%2C%20Liu%2C%20H.%2C%20So%2C%20A.%20M.-C.%2C%20%26amp%3B%20Balzano%2C%20L.%20%282022%29.%20Convergence%20and%20Recovery%20Guarantees%20of%20the%20K-Subspaces%20Method%20for%20Subspace%20Clustering.%20%26lt%3Bi%26gt%3BProceedings%20of%20the%2039th%20International%20Conference%20on%20Machine%20Learning%26lt%3B%5C%2Fi%26gt%3B%2C%2022884%26%23x2013%3B22918.%20%26lt%3Ba%20class%3D%26%23039%3Bzp-ItemURL%26%23039%3B%20href%3D%26%23039%3Bhttps%3A%5C%2F%5C%2Fproceedings.mlr.press%5C%2Fv162%5C%2Fwang22r.html%26%23039%3B%26gt%3Bhttps%3A%5C%2F%5C%2Fproceedings.mlr.press%5C%2Fv162%5C%2Fwang22r.html%26lt%3B%5C%2Fa%26gt%3B%26lt%3B%5C%2Fdiv%26gt%3B%5Cn%26lt%3B%5C%2Fdiv%26gt%3B%22%2C%22data%22%3A%7B%22itemType%22%3A%22conferencePaper%22%2C%22title%22%3A%22Convergence%20and%20Recovery%20Guarantees%20of%20the%20K-Subspaces%20Method%20for%20Subspace%20Clustering%22%2C%22creators%22%3A%5B%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Peng%22%2C%22lastName%22%3A%22Wang%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Huikang%22%2C%22lastName%22%3A%22Liu%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Anthony%20Man-Cho%22%2C%22lastName%22%3A%22So%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Laura%22%2C%22lastName%22%3A%22Balzano%22%7D%5D%2C%22abstractNote%22%3A%22The%20K-subspaces%20%28KSS%29%20method%20is%20a%20generalization%20of%20the%20K-means%20method%20for%20subspace%20clustering.%20In%20this%20work%2C%20we%20present%20local%20convergence%20analysis%20and%20a%20recovery%20guarantee%20for%20KSS%2C%20assuming%20data%20are%20generated%20by%20the%20semi-random%20union%20of%20subspaces%20model%2C%20where%20%5Cud835%5Cudc41NN%20points%20are%20randomly%20sampled%20from%20%5Cud835%5Cudc3e%5Cu22652K%5Cu22652K%20%5C%5Cge%202%20overlapping%20subspaces.%20We%20show%20that%20if%20the%20initial%20assignment%20of%20the%20KSS%20method%20lies%20within%20a%20neighborhood%20of%20a%20true%20clustering%2C%20it%20converges%20at%20a%20superlinear%20rate%20and%20finds%20the%20correct%20clustering%20within%20%5Cu0398%28loglog%5Cud835%5Cudc41%29%5Cu0398%28log%5Cu2061log%5Cu2061N%29%5C%5CTheta%28%5C%5Clog%5C%5Clog%20N%29%20iterations%20with%20high%20probability.%20Moreover%2C%20we%20propose%20a%20thresholding%20inner-product%20based%20spectral%20method%20for%20initialization%20and%20prove%20that%20it%20produces%20a%20point%20in%20this%20neighborhood.%20We%20also%20present%20numerical%20results%20of%20the%20studied%20method%20to%20support%20our%20theoretical%20developments.%22%2C%22proceedingsTitle%22%3A%22Proceedings%20of%20the%2039th%20International%20Conference%20on%20Machine%20Learning%22%2C%22conferenceName%22%3A%22International%20Conference%20on%20Machine%20Learning%22%2C%22date%22%3A%222022-06-28%22%2C%22eventPlace%22%3A%22%22%2C%22DOI%22%3A%22%22%2C%22ISBN%22%3A%22%22%2C%22citationKey%22%3A%22%22%2C%22url%22%3A%22https%3A%5C%2F%5C%2Fproceedings.mlr.press%5C%2Fv162%5C%2Fwang22r.html%22%2C%22ISSN%22%3A%222640-3498%22%2C%22language%22%3A%22en%22%2C%22collections%22%3A%5B%22DZFDBB6V%22%2C%22HJQ26QYG%22%2C%22ZA8QMDGD%22%5D%2C%22dateModified%22%3A%222022-08-10T03%3A40%3A25Z%22%7D%7D%2C%7B%22key%22%3A%22AEIWUC8S%22%2C%22library%22%3A%7B%22id%22%3A1399621%7D%2C%22meta%22%3A%7B%22creatorSummary%22%3A%22Balzano%22%2C%22parsedDate%22%3A%222022-05-03%22%2C%22numChildren%22%3A1%7D%2C%22bib%22%3A%22%26lt%3Bdiv%20class%3D%26quot%3Bcsl-bib-body%26quot%3B%20style%3D%26quot%3Bline-height%3A%202%3B%20padding-left%3A%201em%3B%20text-indent%3A-1em%3B%26quot%3B%26gt%3B%5Cn%20%20%26lt%3Bdiv%20class%3D%26quot%3Bcsl-entry%26quot%3B%26gt%3BBalzano%2C%20L.%20%282022%29.%20On%20the%20equivalence%20of%20Oja%26%23x2019%3Bs%20algorithm%20and%20GROUSE.%20%26lt%3Bi%26gt%3BProceedings%20of%20The%2025th%20International%20Conference%20on%20Artificial%20Intelligence%20and%20Statistics%26lt%3B%5C%2Fi%26gt%3B%2C%207014%26%23x2013%3B7030.%20%26lt%3Ba%20class%3D%26%23039%3Bzp-ItemURL%26%23039%3B%20href%3D%26%23039%3Bhttps%3A%5C%2F%5C%2Fproceedings.mlr.press%5C%2Fv151%5C%2Fbalzano22a.html%26%23039%3B%26gt%3Bhttps%3A%5C%2F%5C%2Fproceedings.mlr.press%5C%2Fv151%5C%2Fbalzano22a.html%26lt%3B%5C%2Fa%26gt%3B%26lt%3B%5C%2Fdiv%26gt%3B%5Cn%26lt%3B%5C%2Fdiv%26gt%3B%22%2C%22data%22%3A%7B%22itemType%22%3A%22conferencePaper%22%2C%22title%22%3A%22On%20the%20equivalence%20of%20Oja%5Cu2019s%20algorithm%20and%20GROUSE%22%2C%22creators%22%3A%5B%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Laura%22%2C%22lastName%22%3A%22Balzano%22%7D%5D%2C%22abstractNote%22%3A%22The%20analysis%20of%20streaming%20PCA%20has%20gained%20significant%20traction%20through%20the%20analysis%20of%20an%20early%20simple%20variant%3A%20Oja%5Cu2019s%20algorithm%2C%20which%20implements%20online%20projected%20gradient%20descent%20for%20the%20trace%20objective.%20Several%20other%20streaming%20PCA%20algorithms%20have%20been%20developed%2C%20each%20with%20their%20own%20performance%20guarantees%20or%20empirical%20studies%2C%20and%20the%20question%20arises%20whether%20there%20is%20a%20relationship%20between%20the%20algorithms.%20We%20show%20that%20the%20Grassmannian%20Rank-One%20Subspace%20Estimation%20%28GROUSE%29%20algorithm%20is%20indeed%20equivalent%20to%20Oja%5Cu2019s%20algorithm%20in%20the%20sense%20that%2C%20at%20each%20iteration%2C%20given%20a%20step%20size%20for%20one%20of%20the%20algorithms%2C%20we%20may%20construct%20a%20step%20size%20for%20the%20other%20algorithm%20that%20results%20in%20an%20identical%20update.%20This%20allows%20us%20to%20apply%20all%20results%20on%20one%20algorithm%20to%20the%20other.%20In%20particular%2C%20we%20have%20%281%29%20better%20global%20convergence%20guarantees%20of%20GROUSE%20to%20the%20global%20minimizer%20of%20the%20PCA%20objective%20with%20full%20data%3B%20and%20%282%29%20local%20convergence%20guarantees%20for%20Oja%5Cu2019s%20algorithm%20with%20incomplete%20or%20compressed%20data.%22%2C%22proceedingsTitle%22%3A%22Proceedings%20of%20The%2025th%20International%20Conference%20on%20Artificial%20Intelligence%20and%20Statistics%22%2C%22conferenceName%22%3A%22International%20Conference%20on%20Artificial%20Intelligence%20and%20Statistics%22%2C%22date%22%3A%222022-05-03%22%2C%22eventPlace%22%3A%22%22%2C%22DOI%22%3A%22%22%2C%22ISBN%22%3A%22%22%2C%22citationKey%22%3A%22%22%2C%22url%22%3A%22https%3A%5C%2F%5C%2Fproceedings.mlr.press%5C%2Fv151%5C%2Fbalzano22a.html%22%2C%22ISSN%22%3A%222640-3498%22%2C%22language%22%3A%22en%22%2C%22collections%22%3A%5B%22DZFDBB6V%22%2C%22HJQ26QYG%22%2C%22ZA8QMDGD%22%5D%2C%22dateModified%22%3A%222022-09-14T14%3A41%3A51Z%22%7D%7D%2C%7B%22key%22%3A%22SHTV82B7%22%2C%22library%22%3A%7B%22id%22%3A1399621%7D%2C%22meta%22%3A%7B%22creatorSummary%22%3A%22Yaras%20et%20al.%22%2C%22parsedDate%22%3A%222022%22%2C%22numChildren%22%3A2%7D%2C%22bib%22%3A%22%26lt%3Bdiv%20class%3D%26quot%3Bcsl-bib-body%26quot%3B%20style%3D%26quot%3Bline-height%3A%202%3B%20padding-left%3A%201em%3B%20text-indent%3A-1em%3B%26quot%3B%26gt%3B%5Cn%20%20%26lt%3Bdiv%20class%3D%26quot%3Bcsl-entry%26quot%3B%26gt%3BYaras%2C%20C.%2C%20Wang%2C%20P.%2C%20Zhu%2C%20Z.%2C%20Balzano%2C%20L.%2C%20%26amp%3B%20Qu%2C%20Q.%20%282022%29.%20%26lt%3Bi%26gt%3BNeural%20Collapse%20with%20Normalized%20Features%3A%20A%20Geometric%20Analysis%20over%20the%20Riemannian%20Manifold%26lt%3B%5C%2Fi%26gt%3B.%20Advances%20in%20Neural%20Information%20Processing%20Systems.%20%26lt%3Ba%20class%3D%26%23039%3Bzp-ItemURL%26%23039%3B%20href%3D%26%23039%3Bhttps%3A%5C%2F%5C%2Fopenreview.net%5C%2Fforum%3Fid%3DZvh6lF5b26N%26%23039%3B%26gt%3Bhttps%3A%5C%2F%5C%2Fopenreview.net%5C%2Fforum%3Fid%3DZvh6lF5b26N%26lt%3B%5C%2Fa%26gt%3B%26lt%3B%5C%2Fdiv%26gt%3B%5Cn%26lt%3B%5C%2Fdiv%26gt%3B%22%2C%22data%22%3A%7B%22itemType%22%3A%22conferencePaper%22%2C%22title%22%3A%22Neural%20Collapse%20with%20Normalized%20Features%3A%20A%20Geometric%20Analysis%20over%20the%20Riemannian%20Manifold%22%2C%22creators%22%3A%5B%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Can%22%2C%22lastName%22%3A%22Yaras%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Peng%22%2C%22lastName%22%3A%22Wang%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Zhihui%22%2C%22lastName%22%3A%22Zhu%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Laura%22%2C%22lastName%22%3A%22Balzano%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Qing%22%2C%22lastName%22%3A%22Qu%22%7D%5D%2C%22abstractNote%22%3A%22When%20training%20overparameterized%20deep%20networks%20for%20classification%20tasks%2C%20it%20has%20been%20widely%20observed%20that%20the%20learned%20features%20exhibit%20a%20so-called%20%26quot%3Bneural%20collapse%26%23039%3B%26quot%3B%20phenomenon.%20More%20specifically%2C%20for%20the%20output%20features%20of%20the%20penultimate%20layer%2C%20for%20each%20class%20the%20within-class%20features%20converge%20to%20their%20means%2C%20and%20the%20means%20of%20different%20classes%20exhibit%20a%20certain%20tight%20frame%20structure%2C%20which%20is%20also%20aligned%20with%20the%20last%20layer%26%23039%3Bs%20classifier.%20As%20feature%20normalization%20in%20the%20last%20layer%20becomes%20a%20common%20practice%20in%20modern%20representation%20learning%2C%20in%20this%20work%20we%20theoretically%20justify%20the%20neural%20collapse%20phenomenon%20under%20normalized%20features.%20Based%20on%20an%20unconstrained%20feature%20model%2C%20we%20simplify%20the%20empirical%20loss%20function%20in%20a%20multi-class%20classification%20task%20into%20a%20nonconvex%20optimization%20problem%20over%20the%20Riemannian%20manifold%20by%20constraining%20all%20features%20and%20classifiers%20over%20the%20sphere.%20In%20this%20context%2C%20we%20analyze%20the%20nonconvex%20landscape%20of%20the%20Riemannian%20optimization%20problem%20over%20the%20product%20of%20spheres%2C%20showing%20a%20benign%20global%20landscape%20in%20the%20sense%20that%20the%20only%20global%20minimizers%20are%20the%20neural%20collapse%20solutions%20while%20all%20other%20critical%20points%20are%20strict%20saddle%20points%20with%20negative%20curvature.%20Experimental%20results%20on%20practical%20deep%20networks%20corroborate%20our%20theory%20and%20demonstrate%20that%20better%20representations%20can%20be%20learned%20faster%20via%20feature%20normalization.%20Code%20for%20our%20experiments%20can%20be%20found%20at%20https%3A%5C%2F%5C%2Fgithub.com%5C%2Fcjyaras%5C%2Fnormalized-neural-collapse.%22%2C%22proceedingsTitle%22%3A%22%22%2C%22conferenceName%22%3A%22Advances%20in%20Neural%20Information%20Processing%20Systems%22%2C%22date%22%3A%222022%22%2C%22eventPlace%22%3A%22%22%2C%22DOI%22%3A%22%22%2C%22ISBN%22%3A%22%22%2C%22citationKey%22%3A%22%22%2C%22url%22%3A%22https%3A%5C%2F%5C%2Fopenreview.net%5C%2Fforum%3Fid%3DZvh6lF5b26N%22%2C%22ISSN%22%3A%22%22%2C%22language%22%3A%22en%22%2C%22collections%22%3A%5B%22DZFDBB6V%22%2C%22HJQ26QYG%22%2C%22ZA8QMDGD%22%5D%2C%22dateModified%22%3A%222022-12-05T21%3A10%3A29Z%22%7D%7D%2C%7B%22key%22%3A%223WT26UID%22%2C%22library%22%3A%7B%22id%22%3A1399621%7D%2C%22meta%22%3A%7B%22creatorSummary%22%3A%22Lipor%20et%20al.%22%2C%22parsedDate%22%3A%222021-03-16%22%2C%22numChildren%22%3A1%7D%2C%22bib%22%3A%22%26lt%3Bdiv%20class%3D%26quot%3Bcsl-bib-body%26quot%3B%20style%3D%26quot%3Bline-height%3A%202%3B%20padding-left%3A%201em%3B%20text-indent%3A-1em%3B%26quot%3B%26gt%3B%5Cn%20%20%26lt%3Bdiv%20class%3D%26quot%3Bcsl-entry%26quot%3B%26gt%3BLipor%2C%20J.%2C%20Hong%2C%20D.%2C%20Tan%2C%20Y.%20S.%2C%20%26amp%3B%20Balzano%2C%20L.%20%282021%29.%20Subspace%20clustering%20using%20ensembles%20of%20K-subspaces.%20%26lt%3Bi%26gt%3BInformation%20and%20Inference%3A%20A%20Journal%20of%20the%20IMA%26lt%3B%5C%2Fi%26gt%3B%2C%20%26lt%3Bi%26gt%3B10%26lt%3B%5C%2Fi%26gt%3B%281%29%2C%2073%26%23x2013%3B107.%20%26lt%3Ba%20class%3D%26%23039%3Bzp-ItemURL%26%23039%3B%20href%3D%26%23039%3Bhttps%3A%5C%2F%5C%2Fdoi.org%5C%2F10.1093%5C%2Fimaiai%5C%2Fiaaa031%26%23039%3B%26gt%3Bhttps%3A%5C%2F%5C%2Fdoi.org%5C%2F10.1093%5C%2Fimaiai%5C%2Fiaaa031%26lt%3B%5C%2Fa%26gt%3B%26lt%3B%5C%2Fdiv%26gt%3B%5Cn%26lt%3B%5C%2Fdiv%26gt%3B%22%2C%22data%22%3A%7B%22itemType%22%3A%22journalArticle%22%2C%22title%22%3A%22Subspace%20clustering%20using%20ensembles%20of%20K-subspaces%22%2C%22creators%22%3A%5B%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22John%22%2C%22lastName%22%3A%22Lipor%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22David%22%2C%22lastName%22%3A%22Hong%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Yan%20Shuo%22%2C%22lastName%22%3A%22Tan%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Laura%22%2C%22lastName%22%3A%22Balzano%22%7D%5D%2C%22abstractNote%22%3A%22Subspace%20clustering%20is%20the%20unsupervised%20grouping%20of%20points%20lying%20near%20a%20union%20of%20low-dimensional%20linear%20subspaces.%20Algorithms%20based%20directly%20on%20geometric%20properties%20of%20such%20data%20tend%20to%20either%20provide%20poor%20empirical%20performance%2C%20lack%20theoretical%20guarantees%20or%20depend%20heavily%20on%20their%20initialization.%20We%20present%20a%20novel%20geometric%20approach%20to%20the%20subspace%20clustering%20problem%20that%20leverages%20ensembles%20of%20the%20%24K%24-subspace%20%28KSS%29%20algorithm%20via%20the%20evidence%20accumulation%20clustering%20framework.%20Our%20algorithm%2C%20referred%20to%20as%20ensemble%20%24K%24-subspaces%20%28EKSSs%29%2C%20forms%20a%20co-association%20matrix%20whose%20%24%28i%2Cj%29%24th%20entry%20is%20the%20number%20of%20times%20points%20%24i%24%20and%20%24j%24%20are%20clustered%20together%20by%20several%20runs%20of%20KSS%20with%20random%20initializations.%20We%20prove%20general%20recovery%20guarantees%20for%20any%20algorithm%20that%20forms%20an%20affinity%20matrix%20with%20entries%20close%20to%20a%20monotonic%20transformation%20of%20pairwise%20absolute%20inner%20products.%20We%20then%20show%20that%20a%20specific%20instance%20of%20EKSS%20results%20in%20an%20affinity%20matrix%20with%20entries%20of%20this%20form%2C%20and%20hence%20our%20proposed%20algorithm%20can%20provably%20recover%20subspaces%20under%20similar%20conditions%20to%20state-of-the-art%20algorithms.%20The%20finding%20is%2C%20to%20the%20best%20of%20our%20knowledge%2C%20the%20first%20recovery%20guarantee%20for%20evidence%20accumulation%20clustering%20and%20for%20KSS%20variants.%20We%20show%20on%20synthetic%20data%20that%20our%20method%20performs%20well%20in%20the%20traditionally%20challenging%20settings%20of%20subspaces%20with%20large%20intersection%2C%20subspaces%20with%20small%20principal%20angles%20and%20noisy%20data.%20Finally%2C%20we%20evaluate%20our%20algorithm%20on%20six%20common%20benchmark%20datasets%20and%20show%20that%20unlike%20existing%20methods%2C%20EKSS%20achieves%20excellent%20empirical%20performance%20when%20there%20are%20both%20a%20small%20and%20large%20number%20of%20points%20per%20subspace.%22%2C%22date%22%3A%22March%2016%2C%202021%22%2C%22section%22%3A%22%22%2C%22partNumber%22%3A%22%22%2C%22partTitle%22%3A%22%22%2C%22DOI%22%3A%2210.1093%5C%2Fimaiai%5C%2Fiaaa031%22%2C%22citationKey%22%3A%22%22%2C%22url%22%3A%22https%3A%5C%2F%5C%2Fdoi.org%5C%2F10.1093%5C%2Fimaiai%5C%2Fiaaa031%22%2C%22PMID%22%3A%22%22%2C%22PMCID%22%3A%22%22%2C%22ISSN%22%3A%222049-8772%22%2C%22language%22%3A%22%22%2C%22collections%22%3A%5B%22427SEM27%22%2C%226JKB3X7P%22%2C%22DZFDBB6V%22%2C%22HJQ26QYG%22%5D%2C%22dateModified%22%3A%222021-03-16T15%3A37%3A52Z%22%7D%7D%2C%7B%22key%22%3A%22VJTA8NLZ%22%2C%22library%22%3A%7B%22id%22%3A1399621%7D%2C%22meta%22%3A%7B%22creatorSummary%22%3A%22Ongie%20et%20al.%22%2C%22parsedDate%22%3A%222021-01-01%22%2C%22numChildren%22%3A2%7D%2C%22bib%22%3A%22%26lt%3Bdiv%20class%3D%26quot%3Bcsl-bib-body%26quot%3B%20style%3D%26quot%3Bline-height%3A%202%3B%20padding-left%3A%201em%3B%20text-indent%3A-1em%3B%26quot%3B%26gt%3B%5Cn%20%20%26lt%3Bdiv%20class%3D%26quot%3Bcsl-entry%26quot%3B%26gt%3BOngie%2C%20G.%2C%20Pimentel-Alarc%26%23xF3%3Bn%2C%20D.%2C%20Balzano%2C%20L.%2C%20Willett%2C%20R.%2C%20%26amp%3B%20Nowak%2C%20R.%20D.%20%282021%29.%20Tensor%20Methods%20for%20Nonlinear%20Matrix%20Completion.%20%26lt%3Bi%26gt%3BSIAM%20Journal%20on%20Mathematics%20of%20Data%20Science%26lt%3B%5C%2Fi%26gt%3B%2C%20253%26%23x2013%3B279.%20%26lt%3Ba%20class%3D%26%23039%3Bzp-DOIURL%26%23039%3B%20href%3D%26%23039%3Bhttps%3A%5C%2F%5C%2Fdoi.org%5C%2F10.1137%5C%2F20M1323448%26%23039%3B%26gt%3Bhttps%3A%5C%2F%5C%2Fdoi.org%5C%2F10.1137%5C%2F20M1323448%26lt%3B%5C%2Fa%26gt%3B%26lt%3B%5C%2Fdiv%26gt%3B%5Cn%26lt%3B%5C%2Fdiv%26gt%3B%22%2C%22data%22%3A%7B%22itemType%22%3A%22journalArticle%22%2C%22title%22%3A%22Tensor%20Methods%20for%20Nonlinear%20Matrix%20Completion%22%2C%22creators%22%3A%5B%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Greg%22%2C%22lastName%22%3A%22Ongie%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Daniel%22%2C%22lastName%22%3A%22Pimentel-Alarc%5Cu00f3n%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Laura%22%2C%22lastName%22%3A%22Balzano%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Rebecca%22%2C%22lastName%22%3A%22Willett%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Robert%20D.%22%2C%22lastName%22%3A%22Nowak%22%7D%5D%2C%22abstractNote%22%3A%22In%20the%20low-rank%20matrix%20completion%20%28LRMC%29%20problem%2C%20the%20low-rank%20assumption%20means%20that%20the%20columns%20%28or%20rows%29%20of%20the%20matrix%20to%20be%20completed%20are%20points%20on%20a%20low-dimensional%20linear%20algebraic%20variety.%20This%20paper%20extends%20this%20thinking%20to%20cases%20where%20the%20columns%20are%20points%20on%20a%20low-dimensional%20nonlinear%20algebraic%20variety%2C%20a%20problem%20we%20call%20low%20algebraic%20dimension%20matrix%20completion%20%28LADMC%29.%20Matrices%20whose%20columns%20belong%20to%20a%20union%20of%20subspaces%20are%20an%20important%20special%20case.%20We%20propose%20an%20LADMC%20algorithm%20that%20leverages%20existing%20LRMC%20methods%20on%20a%20tensorized%20representation%20of%20the%20data.%20For%20example%2C%20a%20second-order%20tensorized%20representation%20is%20formed%20by%20taking%20the%20Kronecker%20product%20of%20each%20column%20with%20itself%2C%20and%20we%20consider%20higher-order%20tensorizations%20as%20well.%20This%20approach%20will%20succeed%20in%20many%20cases%20where%20traditional%20LRMC%20is%20guaranteed%20to%20fail%20because%20the%20data%20are%20low-rank%20in%20the%20tensorized%20representation%20but%20are%20not%20in%20the%20original%20representation.%20We%20also%20provide%20a%20formal%20mathematical%20justification%20for%20the%20success%20of%20our%20method.%20In%20particular%2C%20we%20give%20bounds%20on%20the%20rank%20of%20these%20data%20in%20the%20tensorized%20representation%2C%20and%20we%20prove%20sampling%20requirements%20to%20guarantee%20uniqueness%20of%20the%20solution.%20We%20also%20provide%20experimental%20results%20showing%20that%20the%20new%20approach%20outperforms%20existing%20state-of-the-art%20methods%20for%20matrix%20completion%20under%20a%20union-of-subspaces%20model.%22%2C%22date%22%3A%22January%201%2C%202021%22%2C%22section%22%3A%22%22%2C%22partNumber%22%3A%22%22%2C%22partTitle%22%3A%22%22%2C%22DOI%22%3A%2210.1137%5C%2F20M1323448%22%2C%22citationKey%22%3A%22%22%2C%22url%22%3A%22http%3A%5C%2F%5C%2Fepubs.siam.org%5C%2Fdoi%5C%2Fabs%5C%2F10.1137%5C%2F20M1323448%22%2C%22PMID%22%3A%22%22%2C%22PMCID%22%3A%22%22%2C%22ISSN%22%3A%22%22%2C%22language%22%3A%22%22%2C%22collections%22%3A%5B%22427SEM27%22%2C%22DZFDBB6V%22%2C%22F7QF4T2Q%22%2C%22HJQ26QYG%22%2C%22WUTP7CH6%22%5D%2C%22dateModified%22%3A%222021-03-10T15%3A33%3A01Z%22%7D%7D%2C%7B%22key%22%3A%22EAX6HWJ7%22%2C%22library%22%3A%7B%22id%22%3A1399621%7D%2C%22meta%22%3A%7B%22creatorSummary%22%3A%22Bower%20and%20Balzano%22%2C%22parsedDate%22%3A%222020-11-21%22%2C%22numChildren%22%3A2%7D%2C%22bib%22%3A%22%26lt%3Bdiv%20class%3D%26quot%3Bcsl-bib-body%26quot%3B%20style%3D%26quot%3Bline-height%3A%202%3B%20padding-left%3A%201em%3B%20text-indent%3A-1em%3B%26quot%3B%26gt%3B%5Cn%20%20%26lt%3Bdiv%20class%3D%26quot%3Bcsl-entry%26quot%3B%26gt%3BBower%2C%20A.%2C%20%26amp%3B%20Balzano%2C%20L.%20%282020%29.%20Preference%20Modeling%20with%20Context-Dependent%20Salient%20Features.%20%26lt%3Bi%26gt%3BInternational%20Conference%20on%20Machine%20Learning%26lt%3B%5C%2Fi%26gt%3B%2C%201067%26%23x2013%3B1077.%20%26lt%3Ba%20class%3D%26%23039%3Bzp-ItemURL%26%23039%3B%20href%3D%26%23039%3Bhttps%3A%5C%2F%5C%2Fproceedings.mlr.press%5C%2Fv119%5C%2Fbower20a.html%26%23039%3B%26gt%3Bhttps%3A%5C%2F%5C%2Fproceedings.mlr.press%5C%2Fv119%5C%2Fbower20a.html%26lt%3B%5C%2Fa%26gt%3B%26lt%3B%5C%2Fdiv%26gt%3B%5Cn%26lt%3B%5C%2Fdiv%26gt%3B%22%2C%22data%22%3A%7B%22itemType%22%3A%22conferencePaper%22%2C%22title%22%3A%22Preference%20Modeling%20with%20Context-Dependent%20Salient%20Features%22%2C%22creators%22%3A%5B%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Amanda%22%2C%22lastName%22%3A%22Bower%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Laura%22%2C%22lastName%22%3A%22Balzano%22%7D%5D%2C%22abstractNote%22%3A%22%22%2C%22proceedingsTitle%22%3A%22International%20Conference%20on%20Machine%20Learning%22%2C%22conferenceName%22%3A%22International%20Conference%20on%20Machine%20Learning%22%2C%22date%22%3A%222020-11-21%22%2C%22eventPlace%22%3A%22%22%2C%22DOI%22%3A%22%22%2C%22ISBN%22%3A%22%22%2C%22citationKey%22%3A%22%22%2C%22url%22%3A%22https%3A%5C%2F%5C%2Fproceedings.mlr.press%5C%2Fv119%5C%2Fbower20a.html%22%2C%22ISSN%22%3A%222640-3498%22%2C%22language%22%3A%22en%22%2C%22collections%22%3A%5B%22DZFDBB6V%22%2C%22HJQ26QYG%22%2C%22ZA8QMDGD%22%5D%2C%22dateModified%22%3A%222021-08-25T11%3A19%3A24Z%22%7D%7D%2C%7B%22key%22%3A%22FKPFDDE8%22%2C%22library%22%3A%7B%22id%22%3A1399621%7D%2C%22meta%22%3A%7B%22creatorSummary%22%3A%22Balzano%20et%20al.%22%2C%22parsedDate%22%3A%222018%22%2C%22numChildren%22%3A4%7D%2C%22bib%22%3A%22%26lt%3Bdiv%20class%3D%26quot%3Bcsl-bib-body%26quot%3B%20style%3D%26quot%3Bline-height%3A%202%3B%20padding-left%3A%201em%3B%20text-indent%3A-1em%3B%26quot%3B%26gt%3B%5Cn%20%20%26lt%3Bdiv%20class%3D%26quot%3Bcsl-entry%26quot%3B%26gt%3BBalzano%2C%20L.%2C%20Chi%2C%20Y.%2C%20%26amp%3B%20Lu%2C%20Y.%20M.%20%282018%29.%20Streaming%20PCA%20and%20Subspace%20Tracking%3A%20The%20Missing%20Data%20Case.%20%26lt%3Bi%26gt%3BProceedings%20of%20the%20IEEE%26lt%3B%5C%2Fi%26gt%3B%2C%201%26%23x2013%3B18.%20%26lt%3Ba%20class%3D%26%23039%3Bzp-DOIURL%26%23039%3B%20href%3D%26%23039%3Bhttps%3A%5C%2F%5C%2Fdoi.org%5C%2F10.1109%5C%2FJPROC.2018.2847041%26%23039%3B%26gt%3Bhttps%3A%5C%2F%5C%2Fdoi.org%5C%2F10.1109%5C%2FJPROC.2018.2847041%26lt%3B%5C%2Fa%26gt%3B%26lt%3B%5C%2Fdiv%26gt%3B%5Cn%26lt%3B%5C%2Fdiv%26gt%3B%22%2C%22data%22%3A%7B%22itemType%22%3A%22journalArticle%22%2C%22title%22%3A%22Streaming%20PCA%20and%20Subspace%20Tracking%3A%20The%20Missing%20Data%20Case%22%2C%22creators%22%3A%5B%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22L.%22%2C%22lastName%22%3A%22Balzano%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Y.%22%2C%22lastName%22%3A%22Chi%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Y.%20M.%22%2C%22lastName%22%3A%22Lu%22%7D%5D%2C%22abstractNote%22%3A%22For%20many%20modern%20applications%20in%20science%20and%20engineering%2C%20data%20are%20collected%20in%20a%20streaming%20fashion%20carrying%20time-varying%20information%2C%20and%20practitioners%20need%20to%20process%20them%20with%20a%20limited%20amount%20of%20memory%20and%20computational%20resources%20in%20a%20timely%20manner%20for%20decision%20making.%20This%20often%20is%20coupled%20with%20the%20missing%20data%20problem%2C%20such%20that%20only%20a%20small%20fraction%20of%20data%20attributes%20are%20observed.%20These%20complications%20impose%20significant%2C%20and%20unconventional%2C%20constraints%20on%20the%20problem%20of%20streaming%20principal%20component%20analysis%20%28PCA%29%20and%20subspace%20tracking%2C%20which%20is%20an%20essential%20building%20block%20for%20many%20inference%20tasks%20in%20signal%20processing%20and%20machine%20learning.%20This%20survey%20article%20reviews%20a%20variety%20of%20classical%20and%20recent%20algorithms%20for%20solving%20this%20problem%20with%20low%20computational%20and%20memory%20complexities%2C%20particularly%20those%20applicable%20in%20the%20big%20data%20regime%20with%20missing%20data.%20We%20illustrate%20that%20streaming%20PCA%20and%20subspace%20tracking%20algorithms%20can%20be%20understood%20through%20algebraic%20and%20geometric%20perspectives%2C%20and%20they%20need%20to%20be%20adjusted%20carefully%20to%20handle%20missing%20data.%20Both%20asymptotic%20and%20nonasymptotic%20convergence%20guarantees%20are%20reviewed.%20Finally%2C%20we%20benchmark%20the%20performance%20of%20several%20competitive%20algorithms%20in%20the%20presence%20of%20missing%20data%20for%20both%20well-conditioned%20and%20ill-conditioned%20systems.%22%2C%22date%22%3A%222018%22%2C%22section%22%3A%22%22%2C%22partNumber%22%3A%22%22%2C%22partTitle%22%3A%22%22%2C%22DOI%22%3A%2210.1109%5C%2FJPROC.2018.2847041%22%2C%22citationKey%22%3A%22%22%2C%22url%22%3A%22%22%2C%22PMID%22%3A%22%22%2C%22PMCID%22%3A%22%22%2C%22ISSN%22%3A%220018-9219%22%2C%22language%22%3A%22%22%2C%22collections%22%3A%5B%22427SEM27%22%2C%227V94DTC4%22%2C%22DZFDBB6V%22%2C%22HJQ26QYG%22%5D%2C%22dateModified%22%3A%222024-06-25T14%3A38%3A24Z%22%7D%7D%2C%7B%22key%22%3A%22E5N4EZGZ%22%2C%22library%22%3A%7B%22id%22%3A1399621%7D%2C%22meta%22%3A%7B%22creatorSummary%22%3A%22Kennedy%20et%20al.%22%2C%22parsedDate%22%3A%222016-09%22%2C%22numChildren%22%3A2%7D%2C%22bib%22%3A%22%26lt%3Bdiv%20class%3D%26quot%3Bcsl-bib-body%26quot%3B%20style%3D%26quot%3Bline-height%3A%202%3B%20padding-left%3A%201em%3B%20text-indent%3A-1em%3B%26quot%3B%26gt%3B%5Cn%20%20%26lt%3Bdiv%20class%3D%26quot%3Bcsl-entry%26quot%3B%26gt%3BKennedy%2C%20R.%2C%20Balzano%2C%20L.%2C%20Wright%2C%20S.%20J.%2C%20%26amp%3B%20Taylor%2C%20C.%20J.%20%282016%29.%20Online%20algorithms%20for%20factorization-based%20structure%20from%20motion.%20%26lt%3Bi%26gt%3BComputer%20Vision%20and%20Image%20Understanding%26lt%3B%5C%2Fi%26gt%3B%2C%20%26lt%3Bi%26gt%3B150%26lt%3B%5C%2Fi%26gt%3B%2C%20139%26%23x2013%3B152.%20%26lt%3Ba%20class%3D%26%23039%3Bzp-DOIURL%26%23039%3B%20href%3D%26%23039%3Bhttps%3A%5C%2F%5C%2Fdoi.org%5C%2F10.1016%5C%2Fj.cviu.2016.04.011%26%23039%3B%26gt%3Bhttps%3A%5C%2F%5C%2Fdoi.org%5C%2F10.1016%5C%2Fj.cviu.2016.04.011%26lt%3B%5C%2Fa%26gt%3B%26lt%3B%5C%2Fdiv%26gt%3B%5Cn%26lt%3B%5C%2Fdiv%26gt%3B%22%2C%22data%22%3A%7B%22itemType%22%3A%22journalArticle%22%2C%22title%22%3A%22Online%20algorithms%20for%20factorization-based%20structure%20from%20motion%22%2C%22creators%22%3A%5B%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Ryan%22%2C%22lastName%22%3A%22Kennedy%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Laura%22%2C%22lastName%22%3A%22Balzano%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Stephen%20J.%22%2C%22lastName%22%3A%22Wright%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Camillo%20J.%22%2C%22lastName%22%3A%22Taylor%22%7D%5D%2C%22abstractNote%22%3A%22We%20present%20a%20family%20of%20online%20algorithms%20for%20real-time%20factorization-based%20structure%20from%20motion%2C%20leveraging%20a%20relationship%20between%20the%20incremental%20singular%20value%20decomposition%20and%20recently%20proposed%20methods%20for%20online%20matrix%20completion.%20Our%20methods%20are%20orders%20of%20magnitude%20faster%20than%20previous%20state%20of%20the%20art%2C%20can%20handle%20missing%20data%20and%20a%20variable%20number%20of%20feature%20points%2C%20and%20are%20robust%20to%20noise%20and%20sparse%20outliers.%20We%20demonstrate%20our%20methods%20on%20both%20real%20and%20synthetic%20sequences%20and%20show%20that%20they%20perform%20well%20in%20both%20online%20and%20batch%20settings.%20We%20also%20provide%20an%20implementation%20that%20is%20able%20to%20produce%203D%20models%20in%20real%20time%20using%20a%20laptop%20with%20a%20webcam.%22%2C%22date%22%3A%22September%202016%22%2C%22section%22%3A%22%22%2C%22partNumber%22%3A%22%22%2C%22partTitle%22%3A%22%22%2C%22DOI%22%3A%2210.1016%5C%2Fj.cviu.2016.04.011%22%2C%22citationKey%22%3A%22%22%2C%22url%22%3A%22http%3A%5C%2F%5C%2Fwww.sciencedirect.com%5C%2Fscience%5C%2Farticle%5C%2Fpii%5C%2FS1077314216300364%22%2C%22PMID%22%3A%22%22%2C%22PMCID%22%3A%22%22%2C%22ISSN%22%3A%221077-3142%22%2C%22language%22%3A%22%22%2C%22collections%22%3A%5B%22427SEM27%22%2C%227V94DTC4%22%2C%22DZFDBB6V%22%2C%22HJQ26QYG%22%5D%2C%22dateModified%22%3A%222016-08-09T23%3A20%3A57Z%22%7D%7D%2C%7B%22key%22%3A%22DSKI2PB9%22%2C%22library%22%3A%7B%22id%22%3A1399621%7D%2C%22meta%22%3A%7B%22creatorSummary%22%3A%22Zhang%20and%20Balzano%22%2C%22parsedDate%22%3A%222016%22%2C%22numChildren%22%3A2%7D%2C%22bib%22%3A%22%26lt%3Bdiv%20class%3D%26quot%3Bcsl-bib-body%26quot%3B%20style%3D%26quot%3Bline-height%3A%202%3B%20padding-left%3A%201em%3B%20text-indent%3A-1em%3B%26quot%3B%26gt%3B%5Cn%20%20%26lt%3Bdiv%20class%3D%26quot%3Bcsl-entry%26quot%3B%26gt%3BZhang%2C%20D.%2C%20%26amp%3B%20Balzano%2C%20L.%20%282016%29.%20Global%20Convergence%20of%20a%20Grassmannian%20Gradient%20Descent%20Algorithm%20for%20Subspace%20Estimation.%20%26lt%3Bi%26gt%3BProceedings%20of%20the%2019th%20International%20Conference%20on%20Artificial%20Intelligence%20and%20Statistics%26lt%3B%5C%2Fi%26gt%3B%2C%201460%26%23x2013%3B1468.%20%26lt%3Ba%20class%3D%26%23039%3Bzp-ItemURL%26%23039%3B%20href%3D%26%23039%3Bhttps%3A%5C%2F%5C%2Fproceedings.mlr.press%5C%2Fv51%5C%2Fzhang16b.html%26%23039%3B%26gt%3Bhttps%3A%5C%2F%5C%2Fproceedings.mlr.press%5C%2Fv51%5C%2Fzhang16b.html%26lt%3B%5C%2Fa%26gt%3B%26lt%3B%5C%2Fdiv%26gt%3B%5Cn%26lt%3B%5C%2Fdiv%26gt%3B%22%2C%22data%22%3A%7B%22itemType%22%3A%22conferencePaper%22%2C%22title%22%3A%22Global%20Convergence%20of%20a%20Grassmannian%20Gradient%20Descent%20Algorithm%20for%20Subspace%20Estimation%22%2C%22creators%22%3A%5B%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Dejiao%22%2C%22lastName%22%3A%22Zhang%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Laura%22%2C%22lastName%22%3A%22Balzano%22%7D%5D%2C%22abstractNote%22%3A%22%22%2C%22proceedingsTitle%22%3A%22Proceedings%20of%20the%2019th%20International%20Conference%20on%20Artificial%20Intelligence%20and%20Statistics%22%2C%22conferenceName%22%3A%22Proceedings%20of%20the%2019th%20International%20Conference%20on%20Artificial%20Intelligence%20and%20Statistics%22%2C%22date%22%3A%222016%22%2C%22eventPlace%22%3A%22%22%2C%22DOI%22%3A%22%22%2C%22ISBN%22%3A%22%22%2C%22citationKey%22%3A%22%22%2C%22url%22%3A%22https%3A%5C%2F%5C%2Fproceedings.mlr.press%5C%2Fv51%5C%2Fzhang16b.html%22%2C%22ISSN%22%3A%22%22%2C%22language%22%3A%22%22%2C%22collections%22%3A%5B%227V94DTC4%22%2C%22DZFDBB6V%22%2C%22HJQ26QYG%22%2C%22ZA8QMDGD%22%5D%2C%22dateModified%22%3A%222025-10-22T18%3A54%3A49Z%22%7D%7D%2C%7B%22key%22%3A%22684U7TSZ%22%2C%22library%22%3A%7B%22id%22%3A1399621%7D%2C%22meta%22%3A%7B%22creatorSummary%22%3A%22Balzano%20and%20Wright%22%2C%22parsedDate%22%3A%222014-10-08%22%2C%22numChildren%22%3A2%7D%2C%22bib%22%3A%22%26lt%3Bdiv%20class%3D%26quot%3Bcsl-bib-body%26quot%3B%20style%3D%26quot%3Bline-height%3A%202%3B%20padding-left%3A%201em%3B%20text-indent%3A-1em%3B%26quot%3B%26gt%3B%5Cn%20%20%26lt%3Bdiv%20class%3D%26quot%3Bcsl-entry%26quot%3B%26gt%3BBalzano%2C%20L.%2C%20%26amp%3B%20Wright%2C%20S.%20J.%20%282014%29.%20Local%20Convergence%20of%20an%20Algorithm%20for%20Subspace%20Identification%20from%20Partial%20Data.%20%26lt%3Bi%26gt%3BFoundations%20of%20Computational%20Mathematics%26lt%3B%5C%2Fi%26gt%3B%2C%201%26%23x2013%3B36.%20%26lt%3Ba%20class%3D%26%23039%3Bzp-ItemURL%26%23039%3B%20href%3D%26%23039%3Bhttp%3A%5C%2F%5C%2Flink.springer.com%5C%2Farticle%5C%2F10.1007%5C%2Fs10208-014-9227-7%26%23039%3B%26gt%3Bhttp%3A%5C%2F%5C%2Flink.springer.com%5C%2Farticle%5C%2F10.1007%5C%2Fs10208-014-9227-7%26lt%3B%5C%2Fa%26gt%3B%26lt%3B%5C%2Fdiv%26gt%3B%5Cn%26lt%3B%5C%2Fdiv%26gt%3B%22%2C%22data%22%3A%7B%22itemType%22%3A%22journalArticle%22%2C%22title%22%3A%22Local%20Convergence%20of%20an%20Algorithm%20for%20Subspace%20Identification%20from%20Partial%20Data%22%2C%22creators%22%3A%5B%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Laura%22%2C%22lastName%22%3A%22Balzano%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Stephen%20J.%22%2C%22lastName%22%3A%22Wright%22%7D%5D%2C%22abstractNote%22%3A%22Grassmannian%20rank-one%20update%20subspace%20estimation%20%28GROUSE%29%20is%20an%20iterative%20algorithm%20for%20identifying%20a%20linear%20subspace%20of%20%5Cu211dn%5C%5Cmathbb%20%7BR%7D%5En%20from%20data%20consisting%20of%20partial%20observations%20of%20random%20vectors%20from%20that%20subspace.%20This%20paper%20examines%20local%20convergence%20properties%20of%20GROUSE%2C%20under%20assumptions%20on%20the%20randomness%20of%20the%20observed%20vectors%2C%20the%20randomness%20of%20the%20subset%20of%20elements%20observed%20at%20each%20iteration%2C%20and%20incoherence%20of%20the%20subspace%20with%20the%20coordinate%20directions.%20Convergence%20at%20an%20expected%20linear%20rate%20is%20demonstrated%20under%20certain%20assumptions.%20The%20case%20in%20which%20the%20full%20random%20vector%20is%20revealed%20at%20each%20iteration%20allows%20for%20much%20simpler%20analysis%20and%20is%20also%20described.%20GROUSE%20is%20related%20to%20incremental%20SVD%20methods%20and%20to%20gradient%20projection%20algorithms%20in%20optimization.%22%2C%22date%22%3A%222014%5C%2F10%5C%2F08%22%2C%22section%22%3A%22%22%2C%22partNumber%22%3A%22%22%2C%22partTitle%22%3A%22%22%2C%22DOI%22%3A%22%22%2C%22citationKey%22%3A%22%22%2C%22url%22%3A%22http%3A%5C%2F%5C%2Flink.springer.com%5C%2Farticle%5C%2F10.1007%5C%2Fs10208-014-9227-7%22%2C%22PMID%22%3A%22%22%2C%22PMCID%22%3A%22%22%2C%22ISSN%22%3A%221615-3375%2C%201615-3383%22%2C%22language%22%3A%22en%22%2C%22collections%22%3A%5B%22427SEM27%22%2C%22DZFDBB6V%22%2C%22HJQ26QYG%22%2C%22WUTP7CH6%22%5D%2C%22dateModified%22%3A%222015-09-08T16%3A37%3A09Z%22%7D%7D%2C%7B%22key%22%3A%22Q8DSA89M%22%2C%22library%22%3A%7B%22id%22%3A1399621%7D%2C%22meta%22%3A%7B%22creatorSummary%22%3A%22Balzano%20and%20Wright%22%2C%22parsedDate%22%3A%222013-12%22%2C%22numChildren%22%3A2%7D%2C%22bib%22%3A%22%26lt%3Bdiv%20class%3D%26quot%3Bcsl-bib-body%26quot%3B%20style%3D%26quot%3Bline-height%3A%202%3B%20padding-left%3A%201em%3B%20text-indent%3A-1em%3B%26quot%3B%26gt%3B%5Cn%20%20%26lt%3Bdiv%20class%3D%26quot%3Bcsl-entry%26quot%3B%26gt%3BBalzano%2C%20L.%2C%20%26amp%3B%20Wright%2C%20S.%20J.%20%282013%29.%20On%20GROUSE%20and%20incremental%20SVD.%20%26lt%3Bi%26gt%3B2013%20IEEE%205th%20International%20Workshop%20on%20Computational%20Advances%20in%20Multi-Sensor%20Adaptive%20Processing%20%28CAMSAP%29%26lt%3B%5C%2Fi%26gt%3B%2C%201%26%23x2013%3B4.%20%26lt%3Ba%20class%3D%26%23039%3Bzp-ItemURL%26%23039%3B%20href%3D%26%23039%3Bhttp%3A%5C%2F%5C%2Fieeexplore.ieee.org%5C%2Fxpl%5C%2FarticleDetails.jsp%3Ftp%3D%26amp%3Barnumber%3D6713992%26%23039%3B%26gt%3Bhttp%3A%5C%2F%5C%2Fieeexplore.ieee.org%5C%2Fxpl%5C%2FarticleDetails.jsp%3Ftp%3D%26amp%3Barnumber%3D6713992%26lt%3B%5C%2Fa%26gt%3B%26lt%3B%5C%2Fdiv%26gt%3B%5Cn%26lt%3B%5C%2Fdiv%26gt%3B%22%2C%22data%22%3A%7B%22itemType%22%3A%22conferencePaper%22%2C%22title%22%3A%22On%20GROUSE%20and%20incremental%20SVD%22%2C%22creators%22%3A%5B%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22L.%22%2C%22lastName%22%3A%22Balzano%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22S.J.%22%2C%22lastName%22%3A%22Wright%22%7D%5D%2C%22abstractNote%22%3A%22GROUSE%20%28Grassmannian%20Rank-One%20Update%20Subspace%20Estimation%29%20%5B1%5D%20is%20an%20incremental%20algorithm%20for%20identifying%20a%20subspace%20of%20%5Cu211dn%20from%20a%20sequence%20of%20vectors%20in%20this%20subspace%2C%20where%20only%20a%20subset%20of%20components%20of%20each%20vector%20is%20revealed%20at%20each%20iteration.%20Recent%20analysis%20%5B2%5D%20has%20shown%20that%20GROUSE%20converges%20locally%20at%20an%20expected%20linear%20rate%2C%20under%20certain%20assumptions.%20GROUSE%20has%20a%20similar%20flavor%20to%20the%20incremental%20singular%20value%20decomposition%20algorithm%20%5B4%5D%2C%20which%20updates%20the%20SVD%20of%20a%20matrix%20following%20addition%20of%20a%20single%20column.%20In%20this%20paper%2C%20we%20modify%20the%20incremental%20SVD%20approach%20to%20handle%20missing%20data%2C%20and%20demonstrate%20that%20this%20modified%20approach%20is%20equivalent%20to%20GROUSE%2C%20for%20a%20certain%20choice%20of%20an%20algorithmic%20parameter.%22%2C%22proceedingsTitle%22%3A%222013%20IEEE%205th%20International%20Workshop%20on%20Computational%20Advances%20in%20Multi-Sensor%20Adaptive%20Processing%20%28CAMSAP%29%22%2C%22conferenceName%22%3A%222013%20IEEE%205th%20International%20Workshop%20on%20Computational%20Advances%20in%20Multi-Sensor%20Adaptive%20Processing%20%28CAMSAP%29%22%2C%22date%22%3A%22December%202013%22%2C%22eventPlace%22%3A%22%22%2C%22DOI%22%3A%22%22%2C%22ISBN%22%3A%22%22%2C%22citationKey%22%3A%22%22%2C%22url%22%3A%22http%3A%5C%2F%5C%2Fieeexplore.ieee.org%5C%2Fxpl%5C%2FarticleDetails.jsp%3Ftp%3D%26arnumber%3D6713992%22%2C%22ISSN%22%3A%22%22%2C%22language%22%3A%22%22%2C%22collections%22%3A%5B%227V94DTC4%22%2C%22DZFDBB6V%22%2C%22HJQ26QYG%22%2C%22WUTP7CH6%22%2C%22ZA8QMDGD%22%5D%2C%22dateModified%22%3A%222015-09-08T16%3A39%3A20Z%22%7D%7D%2C%7B%22key%22%3A%22RIZN7IZ9%22%2C%22library%22%3A%7B%22id%22%3A1399621%7D%2C%22meta%22%3A%7B%22creatorSummary%22%3A%22He%20et%20al.%22%2C%22parsedDate%22%3A%222012%22%2C%22numChildren%22%3A2%7D%2C%22bib%22%3A%22%26lt%3Bdiv%20class%3D%26quot%3Bcsl-bib-body%26quot%3B%20style%3D%26quot%3Bline-height%3A%202%3B%20padding-left%3A%201em%3B%20text-indent%3A-1em%3B%26quot%3B%26gt%3B%5Cn%20%20%26lt%3Bdiv%20class%3D%26quot%3Bcsl-entry%26quot%3B%26gt%3BHe%2C%20J.%2C%20Balzano%2C%20L.%2C%20%26amp%3B%20Szlam%2C%20A.%20%282012%29.%20Incremental%20gradient%20on%20the%20Grassmannian%20for%20online%20foreground%20and%20background%20separation%20in%20subsampled%20video.%20%26lt%3Bi%26gt%3BComputer%20Vision%20and%20Pattern%20Recognition%20%28CVPR%29%2C%202012%20IEEE%20Conference%20On%26lt%3B%5C%2Fi%26gt%3B%2C%201568%26%23x2013%3B1575.%20%26lt%3Ba%20class%3D%26%23039%3Bzp-ItemURL%26%23039%3B%20href%3D%26%23039%3Bhttp%3A%5C%2F%5C%2Fieeexplore.ieee.org%5C%2Fxpls%5C%2Fabs_all.jsp%3Farnumber%3D6247848%26%23039%3B%26gt%3Bhttp%3A%5C%2F%5C%2Fieeexplore.ieee.org%5C%2Fxpls%5C%2Fabs_all.jsp%3Farnumber%3D6247848%26lt%3B%5C%2Fa%26gt%3B%26lt%3B%5C%2Fdiv%26gt%3B%5Cn%26lt%3B%5C%2Fdiv%26gt%3B%22%2C%22data%22%3A%7B%22itemType%22%3A%22conferencePaper%22%2C%22title%22%3A%22Incremental%20gradient%20on%20the%20Grassmannian%20for%20online%20foreground%20and%20background%20separation%20in%20subsampled%20video%22%2C%22creators%22%3A%5B%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Jun%22%2C%22lastName%22%3A%22He%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Laura%22%2C%22lastName%22%3A%22Balzano%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Arthur%22%2C%22lastName%22%3A%22Szlam%22%7D%5D%2C%22abstractNote%22%3A%22%22%2C%22proceedingsTitle%22%3A%22Computer%20Vision%20and%20Pattern%20Recognition%20%28CVPR%29%2C%202012%20IEEE%20Conference%20on%22%2C%22conferenceName%22%3A%22%22%2C%22date%22%3A%222012%22%2C%22eventPlace%22%3A%22%22%2C%22DOI%22%3A%22%22%2C%22ISBN%22%3A%22%22%2C%22citationKey%22%3A%22%22%2C%22url%22%3A%22http%3A%5C%2F%5C%2Fieeexplore.ieee.org%5C%2Fxpls%5C%2Fabs_all.jsp%3Farnumber%3D6247848%22%2C%22ISSN%22%3A%22%22%2C%22language%22%3A%22%22%2C%22collections%22%3A%5B%227V94DTC4%22%2C%22DZFDBB6V%22%2C%22HJQ26QYG%22%2C%22ZA8QMDGD%22%5D%2C%22dateModified%22%3A%222024-06-25T14%3A32%3A30Z%22%7D%7D%2C%7B%22key%22%3A%22GTXPRIVJ%22%2C%22library%22%3A%7B%22id%22%3A1399621%7D%2C%22meta%22%3A%7B%22creatorSummary%22%3A%22Eriksson%20et%20al.%22%2C%22parsedDate%22%3A%222012%22%2C%22numChildren%22%3A2%7D%2C%22bib%22%3A%22%26lt%3Bdiv%20class%3D%26quot%3Bcsl-bib-body%26quot%3B%20style%3D%26quot%3Bline-height%3A%202%3B%20padding-left%3A%201em%3B%20text-indent%3A-1em%3B%26quot%3B%26gt%3B%5Cn%20%20%26lt%3Bdiv%20class%3D%26quot%3Bcsl-entry%26quot%3B%26gt%3BEriksson%2C%20B.%2C%20Balzano%2C%20L.%2C%20%26amp%3B%20Nowak%2C%20R.%20%282012%29.%20High%20rank%20matrix%20completion.%20%26lt%3Bi%26gt%3BProc.%20of%20Intl.%20Conf.%20on%20Artificial%20Intell.%20and%20Stat%26lt%3B%5C%2Fi%26gt%3B.%20%26lt%3Ba%20class%3D%26%23039%3Bzp-ItemURL%26%23039%3B%20href%3D%26%23039%3Bhttp%3A%5C%2F%5C%2Fjmlr.csail.mit.edu%5C%2Fproceedings%5C%2Fpapers%5C%2Fv22%5C%2Feriksson12%5C%2Feriksson12.pdf%26%23039%3B%26gt%3Bhttp%3A%5C%2F%5C%2Fjmlr.csail.mit.edu%5C%2Fproceedings%5C%2Fpapers%5C%2Fv22%5C%2Feriksson12%5C%2Feriksson12.pdf%26lt%3B%5C%2Fa%26gt%3B%26lt%3B%5C%2Fdiv%26gt%3B%5Cn%26lt%3B%5C%2Fdiv%26gt%3B%22%2C%22data%22%3A%7B%22itemType%22%3A%22conferencePaper%22%2C%22title%22%3A%22High%20rank%20matrix%20completion%22%2C%22creators%22%3A%5B%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Brian%22%2C%22lastName%22%3A%22Eriksson%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Laura%22%2C%22lastName%22%3A%22Balzano%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Robert%22%2C%22lastName%22%3A%22Nowak%22%7D%5D%2C%22abstractNote%22%3A%22%22%2C%22proceedingsTitle%22%3A%22Proc.%20of%20Intl.%20Conf.%20on%20Artificial%20Intell.%20and%20Stat%22%2C%22conferenceName%22%3A%22%22%2C%22date%22%3A%222012%22%2C%22eventPlace%22%3A%22%22%2C%22DOI%22%3A%22%22%2C%22ISBN%22%3A%22%22%2C%22citationKey%22%3A%22%22%2C%22url%22%3A%22http%3A%5C%2F%5C%2Fjmlr.csail.mit.edu%5C%2Fproceedings%5C%2Fpapers%5C%2Fv22%5C%2Feriksson12%5C%2Feriksson12.pdf%22%2C%22ISSN%22%3A%22%22%2C%22language%22%3A%22%22%2C%22collections%22%3A%5B%226JKB3X7P%22%2C%22DZFDBB6V%22%2C%22HJQ26QYG%22%2C%22WUTP7CH6%22%2C%22ZA8QMDGD%22%5D%2C%22dateModified%22%3A%222013-05-17T18%3A38%3A09Z%22%7D%7D%2C%7B%22key%22%3A%22IC5THMK3%22%2C%22library%22%3A%7B%22id%22%3A1399621%7D%2C%22meta%22%3A%7B%22creatorSummary%22%3A%22Balzano%20et%20al.%22%2C%22parsedDate%22%3A%222010%22%2C%22numChildren%22%3A2%7D%2C%22bib%22%3A%22%26lt%3Bdiv%20class%3D%26quot%3Bcsl-bib-body%26quot%3B%20style%3D%26quot%3Bline-height%3A%202%3B%20padding-left%3A%201em%3B%20text-indent%3A-1em%3B%26quot%3B%26gt%3B%5Cn%20%20%26lt%3Bdiv%20class%3D%26quot%3Bcsl-entry%26quot%3B%26gt%3BBalzano%2C%20L.%2C%20Nowak%2C%20R.%2C%20%26amp%3B%20Recht%2C%20B.%20%282010%29.%20Online%20identification%20and%20tracking%20of%20subspaces%20from%20highly%20incomplete%20information.%20%26lt%3Bi%26gt%3BCommunication%2C%20Control%2C%20and%20Computing%20%28Allerton%29%2C%202010%2048th%20Annual%20Allerton%20Conference%20On%26lt%3B%5C%2Fi%26gt%3B%2C%20704%26%23x2013%3B711.%20%26lt%3Ba%20class%3D%26%23039%3Bzp-ItemURL%26%23039%3B%20href%3D%26%23039%3Bhttp%3A%5C%2F%5C%2Fieeexplore.ieee.org%5C%2Fxpls%5C%2Fabs_all.jsp%3Farnumber%3D5706976%26%23039%3B%26gt%3Bhttp%3A%5C%2F%5C%2Fieeexplore.ieee.org%5C%2Fxpls%5C%2Fabs_all.jsp%3Farnumber%3D5706976%26lt%3B%5C%2Fa%26gt%3B%26lt%3B%5C%2Fdiv%26gt%3B%5Cn%26lt%3B%5C%2Fdiv%26gt%3B%22%2C%22data%22%3A%7B%22itemType%22%3A%22conferencePaper%22%2C%22title%22%3A%22Online%20identification%20and%20tracking%20of%20subspaces%20from%20highly%20incomplete%20information%22%2C%22creators%22%3A%5B%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Laura%22%2C%22lastName%22%3A%22Balzano%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Robert%22%2C%22lastName%22%3A%22Nowak%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Benjamin%22%2C%22lastName%22%3A%22Recht%22%7D%5D%2C%22abstractNote%22%3A%22%22%2C%22proceedingsTitle%22%3A%22Communication%2C%20Control%2C%20and%20Computing%20%28Allerton%29%2C%202010%2048th%20Annual%20Allerton%20Conference%20on%22%2C%22conferenceName%22%3A%22%22%2C%22date%22%3A%222010%22%2C%22eventPlace%22%3A%22%22%2C%22DOI%22%3A%22%22%2C%22ISBN%22%3A%22%22%2C%22citationKey%22%3A%22%22%2C%22url%22%3A%22http%3A%5C%2F%5C%2Fieeexplore.ieee.org%5C%2Fxpls%5C%2Fabs_all.jsp%3Farnumber%3D5706976%22%2C%22ISSN%22%3A%22%22%2C%22language%22%3A%22%22%2C%22collections%22%3A%5B%227V94DTC4%22%2C%22DZFDBB6V%22%2C%22HJQ26QYG%22%2C%22WUTP7CH6%22%2C%22ZA8QMDGD%22%5D%2C%22dateModified%22%3A%222013-05-13T21%3A09%3A51Z%22%7D%7D%2C%7B%22key%22%3A%228APWRX4C%22%2C%22library%22%3A%7B%22id%22%3A1399621%7D%2C%22meta%22%3A%7B%22creatorSummary%22%3A%22Balzano%20and%20Nowak%22%2C%22parsedDate%22%3A%222007%22%2C%22numChildren%22%3A2%7D%2C%22bib%22%3A%22%26lt%3Bdiv%20class%3D%26quot%3Bcsl-bib-body%26quot%3B%20style%3D%26quot%3Bline-height%3A%202%3B%20padding-left%3A%201em%3B%20text-indent%3A-1em%3B%26quot%3B%26gt%3B%5Cn%20%20%26lt%3Bdiv%20class%3D%26quot%3Bcsl-entry%26quot%3B%26gt%3BBalzano%2C%20L.%2C%20%26amp%3B%20Nowak%2C%20R.%20%282007%29.%20Blind%20calibration%20of%20sensor%20networks.%20%26lt%3Bi%26gt%3BProceedings%20of%20the%206th%20International%20Conference%20on%20Information%20Processing%20in%20Sensor%20Networks%26lt%3B%5C%2Fi%26gt%3B%2C%2079%26%23x2013%3B88.%20%26lt%3Ba%20class%3D%26%23039%3Bzp-ItemURL%26%23039%3B%20href%3D%26%23039%3Bhttp%3A%5C%2F%5C%2Fdl.acm.org%5C%2Fcitation.cfm%3Fid%3D1236372%26%23039%3B%26gt%3Bhttp%3A%5C%2F%5C%2Fdl.acm.org%5C%2Fcitation.cfm%3Fid%3D1236372%26lt%3B%5C%2Fa%26gt%3B%26lt%3B%5C%2Fdiv%26gt%3B%5Cn%26lt%3B%5C%2Fdiv%26gt%3B%22%2C%22data%22%3A%7B%22itemType%22%3A%22conferencePaper%22%2C%22title%22%3A%22Blind%20calibration%20of%20sensor%20networks%22%2C%22creators%22%3A%5B%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Laura%22%2C%22lastName%22%3A%22Balzano%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Robert%22%2C%22lastName%22%3A%22Nowak%22%7D%5D%2C%22abstractNote%22%3A%22%22%2C%22proceedingsTitle%22%3A%22Proceedings%20of%20the%206th%20international%20conference%20on%20Information%20processing%20in%20sensor%20networks%22%2C%22conferenceName%22%3A%22%22%2C%22date%22%3A%222007%22%2C%22eventPlace%22%3A%22%22%2C%22DOI%22%3A%22%22%2C%22ISBN%22%3A%22%22%2C%22citationKey%22%3A%22%22%2C%22url%22%3A%22http%3A%5C%2F%5C%2Fdl.acm.org%5C%2Fcitation.cfm%3Fid%3D1236372%22%2C%22ISSN%22%3A%22%22%2C%22language%22%3A%22%22%2C%22collections%22%3A%5B%2284TPD646%22%2C%22DZFDBB6V%22%2C%22HJQ26QYG%22%2C%22UIWU664R%22%2C%22ZA8QMDGD%22%5D%2C%22dateModified%22%3A%222013-05-13T21%3A08%3A58Z%22%7D%7D%5D%7D
Hedayat, A., Balzano, L., & Duraisamy, K. (2026). In-span learning: adapting reduced-order models using their own predictions (arXiv:2607.02937). arXiv. https://doi.org/10.48550/arXiv.2607.02937
Balzano, L., Ding, T., Haeffele, B. D., Kwon, S. M., Qu, Q., Wang, P., Wang, Z., & Yaras, C. (2026). An Overview of Low-Rank Structures in the Training and Adaptation of Large Models: From Implicit Low-Dimensionality to Efficient Training and Fine-Tuning [Special Issue on the Mathematics of Deep Learning]. IEEE Signal Processing Magazine, 43(3), 52–70. https://doi.org/10.1109/MSP.2026.3666749
Yaras, C., Xu, A., Abillama, P., Lee, C., & Balzano, L. (2025). MonarchAttention: Zero-Shot Conversion to Fast, Hardware-Aware Structured Attention. Advances in Neural Information Processing Systems, 38, 3445–3471. https://papers.nips.cc/paper_files/paper/2025/hash/050f8591be3874b52fdac4e1060eeb29-Abstract-Conference.html
Gilman, K., Burer, S., & Balzano, L. (2025). A Semidefinite Relaxation for Sums of Heterogeneous Quadratic Forms on the Stiefel Manifold. SIAM Journal on Matrix Analysis and Applications, 1091–1116. https://doi.org/10.1137/23M1545136
Yaras, C., Wang, P., Balzano, L., & Qu, Q. (2024, June 6). Compressible Dynamics in Deep Overparameterized Low-Rank Learning & Adaptation. Forty-first International Conference on Machine Learning. https://openreview.net/forum?id=uDkXoZMzBv
Liu, H., Wang, P., Huang, L., Qu, Q., & Balzano, L. (2024, June 6). Symmetric Matrix Completion with ReLU Sampling. Forty-first International Conference on Machine Learning. https://openreview.net/forum?id=VxI0gInNlh
Kwon, S. M., Zhang, Z., Song, D., Balzano, L., & Qu, Q. (2024). Efficient Low-Dimensional Compression of Overparameterized Models. Proceedings of The 27th International Conference on Artificial Intelligence and Statistics, 1009–1017. https://proceedings.mlr.press/v238/min-kwon24a.html
Hong, D., Yang, F., Fessler, J. A., & Balzano, L. (2023). Optimally Weighted PCA for High-Dimensional Heteroscedastic Data. SIAM Journal on Mathematics of Data Science, 5(1), 222–250. https://doi.org/10.1137/22M1470244
Wang, P., Liu, H., So, A. M.-C., & Balzano, L. (2022). Convergence and Recovery Guarantees of the K-Subspaces Method for Subspace Clustering. Proceedings of the 39th International Conference on Machine Learning, 22884–22918. https://proceedings.mlr.press/v162/wang22r.html
Balzano, L. (2022). On the equivalence of Oja’s algorithm and GROUSE. Proceedings of The 25th International Conference on Artificial Intelligence and Statistics, 7014–7030. https://proceedings.mlr.press/v151/balzano22a.html
Yaras, C., Wang, P., Zhu, Z., Balzano, L., & Qu, Q. (2022). Neural Collapse with Normalized Features: A Geometric Analysis over the Riemannian Manifold. Advances in Neural Information Processing Systems. https://openreview.net/forum?id=Zvh6lF5b26N
Lipor, J., Hong, D., Tan, Y. S., & Balzano, L. (2021). Subspace clustering using ensembles of K-subspaces. Information and Inference: A Journal of the IMA, 10(1), 73–107. https://doi.org/10.1093/imaiai/iaaa031
Ongie, G., Pimentel-Alarcón, D., Balzano, L., Willett, R., & Nowak, R. D. (2021). Tensor Methods for Nonlinear Matrix Completion. SIAM Journal on Mathematics of Data Science, 253–279. https://doi.org/10.1137/20M1323448
Bower, A., & Balzano, L. (2020). Preference Modeling with Context-Dependent Salient Features. International Conference on Machine Learning, 1067–1077. https://proceedings.mlr.press/v119/bower20a.html
Balzano, L., Chi, Y., & Lu, Y. M. (2018). Streaming PCA and Subspace Tracking: The Missing Data Case. Proceedings of the IEEE, 1–18. https://doi.org/10.1109/JPROC.2018.2847041
Kennedy, R., Balzano, L., Wright, S. J., & Taylor, C. J. (2016). Online algorithms for factorization-based structure from motion. Computer Vision and Image Understanding, 150, 139–152. https://doi.org/10.1016/j.cviu.2016.04.011
Zhang, D., & Balzano, L. (2016). Global Convergence of a Grassmannian Gradient Descent Algorithm for Subspace Estimation. Proceedings of the 19th International Conference on Artificial Intelligence and Statistics, 1460–1468. https://proceedings.mlr.press/v51/zhang16b.html
Balzano, L., & Wright, S. J. (2014). Local Convergence of an Algorithm for Subspace Identification from Partial Data. Foundations of Computational Mathematics, 1–36. http://link.springer.com/article/10.1007/s10208-014-9227-7
Balzano, L., & Wright, S. J. (2013). On GROUSE and incremental SVD. 2013 IEEE 5th International Workshop on Computational Advances in Multi-Sensor Adaptive Processing (CAMSAP), 1–4. http://ieeexplore.ieee.org/xpl/articleDetails.jsp?tp=&arnumber=6713992
He, J., Balzano, L., & Szlam, A. (2012). Incremental gradient on the Grassmannian for online foreground and background separation in subsampled video. Computer Vision and Pattern Recognition (CVPR), 2012 IEEE Conference On, 1568–1575. http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=6247848
Eriksson, B., Balzano, L., & Nowak, R. (2012). High rank matrix completion. Proc. of Intl. Conf. on Artificial Intell. and Stat. http://jmlr.csail.mit.edu/proceedings/papers/v22/eriksson12/eriksson12.pdf
Balzano, L., Nowak, R., & Recht, B. (2010). Online identification and tracking of subspaces from highly incomplete information. Communication, Control, and Computing (Allerton), 2010 48th Annual Allerton Conference On, 704–711. http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=5706976
Balzano, L., & Nowak, R. (2007). Blind calibration of sensor networks. Proceedings of the 6th International Conference on Information Processing in Sensor Networks, 79–88. http://dl.acm.org/citation.cfm?id=1236372