EECS 553 F26 Class Topics - Jeff Fessler This list will be updated regularly (online) over the course of the semester. Class# topic @01 9/01 Course policies Classification overview KNN classifier definition colab demo decision boundaries choosing k: underfit vs overfit generative vs discriminative parametric vs nonparametric @02 9/03 computation / scalability Probabilistic methods loss and risk 0-1 loss and error probability surrogate loss preview [skip] Bayes Classifier Bayes risk Binary case Likelihood ratio @03 9/08 Plug-in classifier Linear discriminant analysis (LDA) Data model Maximum-likelihood parameter estimates LDA plug-in / linearity colab demo generative vs discriminative parametric vs nonparametric @04 9/10 LDA demo with digits 1,7 Project overview - reproducibility challenge Quadratic discriminant analysis (QDA) QDA plug-in Iris data example Naive Bayes data model / plug-in estimates document classification example @05 9/15 Regression squared error loss MSE risk conditional mean estimator MMSE estimator - Gaussian case empirical risk / MSE ordinary least squares regularized least squares Tikhonov, LASSO, elastic net Empirical risk minimization (ERM) preview Regularized ERM Surrogate losses for ERM @06 9/17 Optimization preview preview of logistic regression ridge regression compute complexity Unconstrained optimization fundamentals Gradient Taylor expansion Lipschitz continuity Smooth functions 1st-order approximation Local and global minimizers 1st-order necessary condition stationary point, critical point Hessian 2nd-order necessary condition Convex functions @07 9/22 local and global minimizers, saddle points Optimization algorithms Gradient descent ridge regression example Convergence rates Descent methods Descent direction Gradient descent Fixed point / stationary point @08 9/24 Newton's method (2nd order) quadratic example todo: need non-quadratic example todo: quasi-Newton? Majorize-Minimize (MM) for bounded Hessian Convergence: global vs local GD convergence for quadratic cost Newton's method convergence Stochastic gradient descent (SGD) Subgradient method Stochastic subgradient method Logistic regression plug-in estimate of P{Y = 1 | X = x} @09 9/29 log-likelihood regularization demo sparse logistic regression (brief) softmax function / cross entropy multiclass logistic regression imbalanced data Separating hyperplanes @10 10/01 distance to hyperplane ... Max-margin hyperplanes Optimal soft-margin hyperplane: prelude Constrained optimization Lagrangian Duality KKT conditions: necessity @11 10/06 KKT conditions: sufficiency Optimal soft-margin hyperplane: finale @12 10/08 Exam review? [no] Kernel methods Nonlinear lifting Kernel SVM @xx 10/12 (Mon) Exam1 on HW 1-4 and Lecture 1-9 @13 10/13 Kernel nearest neighbor Kernel ridge regression Neural network (NN) models Feedforward networks Project papers feedback [not in f26] @14 10/15 Backpropagation Convolutional neural network (CNN) models Convolution Pooling @xx 10/20 fall study break (may need to adjust HW due date) @15 10/22 Feature learning Self-supervised learning Pretext tasks Word prediction / GPT Sequence modeling Transformer / Attention Motivation @16 10/27 Self-attention model Hyperparameters Overfitting Estimating risk Train / test data Selecting strategies @17 10/29 Holdout Bayesian optimization Derivative-free optimization Black-box optimization Cross validation @18 11/03 Unsupervised learning Representations Embedding Feature extraction Association/clustering/segmentation Density estimation Principal component analysis (PCA) @19 11/05 PCA demo Embedding and dimensionality reduction Nonlinear dimensionality reduction t-SNE Clustering k-Means Lloyd's algorithm @20 11/10 k-means example Gaussian mixture model (GMM) Latent variable models Expectation maximization (EM) --- below here list topics from W26, except with F26 dates --- the topics and due dates are all likely to change @21 11/12 EM applied to GMM @ [was 17] Exam review? [no] Decision trees Classification trees Regression trees Ensemble methods Random forests Boosting Gradient boosting @xx 11/16 (Mon) Exam2 on HW 1-7 and Lecture 1-18 @22 11/17 @ [was 18] Matrix factorization Factor analysis Matrix completion Low-rank matrix approximation Low-rank matrix approximation theory Coherence Low-rank adaptation (LoRA) LRMR results for Gaussian @23 11/19 @ [was 19] Low-rank adaptation (LoRA) Implicit bias Deep factorization Implicit bias of dropout @24 11/24 @ [was 20] @ [was 21] Socially responsible ML Bias Fairness Deepfakes Interpretability Adversarial attacks Current topics @xx 11/26 thanksgiving day no HW due to break @25 12/01 ++ hongze nerf/inr ? ++ score-based diffusion models? @26 12/03 @ EECS 553 retrospective ? ML trends ? @27 12/08 presentations! @27.5 12/09 [evening] presentations! @28 12/10 [last class] presentations! @xx 12/17 (Thu.) final presentations 4-6PM (official RO time) [NO!]