EECS 598: Statistical Learning Theory

University of Michigan, Winter 2014

Instructor: Clayton Scott (clayscot)
Classroom: EECS 1003
Time: MW 9-10:30
Office: 4433 EECS
Office hours: Monday 1-2 or by appt.

Prerequisites:

• Probability at the level of EECS 501 or equivalent
• Experience with formal mathematical proofs.

Recommended books (on reserve at Engineering library):

• Devroye, Gyorfi, and Lugosi, A Probabilistic Theory of Pattern Recognition, Springer, 1996.
• Mohri, Rostamizadeh, and Talwalkar, Foundations of Machine Learning, MIT Press, 2012.

Surveys and tutorials:

• Olivier Bousquet, Stephane Boucheron, and Gabor Lugosi, Introduction to Statistical Learning Theory, in O. Bousquet, U.v. Luxburg, and G. Ratsch (editors), Advanced Lectures in Machine Learning, Springer, pp. 169--207, 2004.
• Ambuj Tewari and Peter L. Bartlett, Learning Theory, in Rama Chellappa and Sergios Theodoridis (editors), Academic Press Library in Signal Processing, volume 1, chapter 14. Elsevier, 1st edition, 2013.

Lecture notes

1. Probabilistic setting
2. The Bayes classifier
3. Hoeffding's inequality
4. Empirical risk minimization
5. Vapnik-Chervonenkis Theory. VC classes, Sauer's lemma, DKW theorem, monotone layers and convex sets.
6. Sieve Estimators: Consistency and Rates of Convergence
7. Dyadic Decision Trees
8. Oracle Inequalities and Adaptive Rates. Structural risk minimization. Adapting to relevant features, intrinsic dimenion.
9. The Bounded Difference Inequality
10. Rademacher Complexity. Proof of VC inequality.
11. Kernels
12. Reproducing Kernel Hilbert Spaces
13. Kernel Methods and the Representer Theorem
14. Calibrated Surrogate Losses
15. Rademacher Complexity of Kernel Classes
16. Margin Bounds
17. Universal Consistency of Support Vector Machines and other Kernel Methods
18. Rates for Linear SVMs under the Hard Margin Assumption
19. Kernel Density Estimation
20. Weakly Supervised Learning, Anomaly detection, classification with label noise.

Grading
Homework (40%)
Final report (40%)
Participation (20%)

Homework
Homework will be assigned progressively in lecture, and will be due at regular intervals. You will be given at least one week advanced notice of the due date, but I recommend solving the problems as they are assigned, since it will help with your understanding of the lectures.

Final report
Each student will choose one or a few research papers on a particular topic in statistical learning theory, and write a report that summarizes the contributions of the paper(s), including at least a sketch of the main technical ideas. Reports will be evaluated by your peers in a manner that mimics a conference review process. I may be out of town the last week of classes, so the reports may be due slightly before then, such as the last Friday before classes end.

Participation
Attendance, classroom interaction, and scribing lecture notes.

Honor Code
All undergraduate and graduate students are expected to abide by the College of Engineering Honor Code as stated in the Student Handbook and the Honor Code Pamphlet.

Students with Disabilities
Any student with a documented disability needing academic adjustments or accommodations is requested to speak with me during the first two weeks of class. All discussions will remain confidential.