Topics and Course Structure

The first half of the course will cover the fundamental components that drive modern deep learning systems for computer vision:

  • Linear classifiers
  • Stochastic gradient descent
  • Fully-connected networks
  • Convolutional networks
  • Recurrent networks

In the second half of the course we will discuss applications of deep learning to different problems in computer vision, as well as more emerging topics. During this second half the tone of the course will shift slightly towards a seminar: we will omit some details of the systems we discuss, instead focusing on the core concepts behind those applications. We will touch topics such as:

  • Attention and transformers
  • Object detection
  • Image segmentation
  • Video classification
  • Generative models (GANs, VAEs, autoregressive models)
  • Deep reinforcement learning

Course Expectations

Deep Learning is a fast-moving, empirically-driven research field. Much of the content we will cover is taken from research papers published in the last 5 to 10 years. It is also a large and fast-growing field of research: there are thousands of research papers published each year on computer vision, deep learning, and related topics. It is possible that we will discuss some new paper or result that will be published or announced during the semester!

Learning about such a fast-moving field is an exciting opportunity, but covering it in a traditional course setting comes with some caveats you should be aware of.

  • No canonical curriculum: Many topics in mathematics and computer science such as linear algebra, real analysis, discrete mathematics, data structures and algorithms, etc come with well-established curricula; courses on such subjects can be found at most universities, and they tend to cover similar topics in a similar order. This is not the case for emerging research areas like deep learning: the set of topics to be covered, as well as the order and way of thinking about each topic, has not yet been perfected.
  • Few learning materials: There are very few high-quality textbooks or other learning materials that synthesize or explain much of the content we will cover. In many cases the research paper that introduced an idea is the best or only resource for learning about it.
  • Theory lags experiments: At present, deep learning is primarily an empirically driven research field. We may use mathematical notation to describe or communicate our algorithms and ideas, and many techniques are motivated by some mathematical or computational intuition, but in most cases we rely on experiments rather than formal proofs to determine the scenarios where one technique might outpeform another. This can sometimes be unsettling for students, as the question “why does that work?” may not always have a precise, theoretically-grounded answer.
  • Things will change: If you were to study deep learning ten years from now, it is very likely that it will look quite different from today. There may be new fundamental discoveries, or new ways of thinking about things we already know; there may be some ideas we think are important today, that will turn out in retrospect not to have been. As a concrete example, when we first taught a similar course at Stanford in 2015, Batch Normalization and Residual Networks had not yet been discovered; these have since become hugely important, and are used in nearly all practical computer vision systems. There may be similarly impactful results lurking right around the corner.

Prerequisites

The following skills will be useful for success in this course:

  • Programming: You should be comfortable programming in Python. You should be familiar with algorithms and data structures at the level of EECS 281 or equivalent. Familiarity with numpy or similar frameworks for numeric programming will be helpful but is not strictly required.
  • Probability: You should have been exposed to probability distributions, random variables, expectations, etc. at the level of EECS 401, MATH 425, or equivalent.
  • Vector Calculus: You will need to compute gradients of vector-valued functions, so you should be comfortable doing so.
  • Machine Learning: Some familiarity with machine learning (at the level of EECS 445 or equivalent) will be helpful but not required; we will review important concepts that are needed for this course.

Textbooks

There is no required textbook for the course. Some lectures will include suggested reading from “Deep Learning” by Goodfellow, Bengio, and Courville; however there is no need to purchase a copy since it is available for free online.

Homework Assignments

You will complete six programming assignments over the course of the semester. All homework assignments will be in Python, and will use PyTorch on Google Colab.

Course Communication

The primary way to communicate with the course staff is through Piazza. If you have questions about course concepts, need help with homework, or have questions about course logistics, you should post on Piazza instead of emailing course staff directly. Since Piazza is a shared discussion forum, asking and answering questions there can benefit other students as well.

However do not post solutions to homework assignments on Piazza. If you have questions about a particular piece of code you have written for an assignment, then you should make a private post on Piazza.

We will also use Piazza to make announcments about the course, such as the release of new homework assignments, details about the midterm exam, or other logistical issues.

If you need to discuss any sensitive matter that you would prefer not be shared with the entire course staff then you can email the instructor directly. However the vast majority of questions about the course should be asked on Piazza.

Grading Policy

  • Homework (75%): There will be six programming assignments. The first assignment is worth 10% of your final grade; each of the other five assignments are worth 13% of your final grade.
  • Exams (25%): The midterm exam will be worth 25% of your final grade.

We will use the following base scheme to convert raw scores in the course into letter grades:

  • A+: >= 98%
  • A: >= 92%
  • A-: >= 90%
  • B+: >= 88%
  • B: >= 82%
  • B-: >= 80%
  • C+: >= 78%
  • C: >= 72%
  • C-: >= 70%
  • D+: >= 68%
  • D: >= 62%
  • D-: >= 60%
  • F: < 60%

Depending on the distribution of raw scores at the end of the semester, we may apply a curve; however the final curve will be no stricter than the base scheme. For example, if you receive a raw score of 85% in the course, you will be guaranteed a letter grade of at least B; however you may receive a higher letter grade depending on the overall distribution of raw scores.

Late Policy

Homework assignments will be turned in on Canvas, and are due at 11:59pm ET on their assigned due date.

Each student has a total of three free late days to use on assignments throughout the semester. Each late day allows you to turn in a single homework assignment up to 24 hours past the initial deadline with no penalty; there is no restriction on the number of free late days you may use per assignment. You do not need permission to use free late days; simply turn in the assignment when you are done, and we will note that you used a free late day when the graded assignment is returned to you.

Once you have exhausted your three free late days, late work will receive a penalty of 25% per day. Again, there is no need to obtain permission in order to turn in late work – we will apply the penalty when grading, and it will be noted when we return the assignment to you. You cannot receive less than 0% on an assignment no matter how many days late it is.

Example:

  • Assignment 1 is due on Friday at 11:59pm; you turn it in at 3am on Sunday. The assignment is two days late. This exhausts two of your three free late days, and you receive no penalty of the assignment.
  • Assignment 2 is due the next Friday at 11:59pm, and you turn it in at 10pm on Monday. The assignment is three days late. This exhausts your last remaining free late day, so you receive a penalty of 50% on the assignment. We grade the assignment as usual, and suppose you recieve a score of 88/100; after applying the late penalty your final recorded score for the assignment is 38/100.
  • Assignment 3 is due the next Friday at 11:59pm, and you turn it on 5pm on Monday. The assignment is three days late, and you receive a late penalty of 75%. We grade the assignment as usual, and suppose you receive a score of 65/100; after applying the late penalty your final recorded score for the assignment is 0/100.

When determining late days, we will use the last revision time of the entire assignment. This means that if you turn in a partially completed assignment before the due date, and then turn in a revised, completed assignment a day after the due date, the entire assignment will be treated as if it had been turned in a day past the deadline.

We will not grant additional late days except in extreme circumstances (serious illness, family tragedy, etc.) Please see the instructor if such a situation arises. In particular we will not grant additional late due to deadlines in other courses, or academic publication deadlines, or travel to academic conferences, or other such events that you can plan ahead for at the start of the semester.

Collaboration Policy

You are encouraged to work in groups and discuss course materials with your classmates. However, all work you submit must be your own.

The following are examples of acceptable collaboration:

  • Forming a study group to discuss and review course concepts
  • Discussing general strategies for solving homework problems
  • Reviewing for exams with other students

The following are examples of unacceptable collaboration:

  • Turning in the same code as another student
  • Looking at another student’s code while writing your own
  • Looking at solutions to similar assignments that you find online