From this assignment forward, you will use autograd in PyTorch to perform backpropgation for you. This will enable you to easily build complex models without worrying about writing code for the backward pass by hand.

The goals of this assignment are:

Understand how autograd can help automate gradient computation
See how to use PyTorch Modules to build up complex neural network architectures
Understand and implement recurrent neural networks
See how recurrent neural networks can be used for image captioning
Understand how to augment recurrent neural networks with attention
Use image gradients to synthesize saliency maps, adversarial examples, and perform class visualizations
Combine content and style losses to perform artistic style transfer

This assignment is due on Wednesday, November 13 at 11:59pm EDT.

Q1: PyTorch Autograd (30 points)

The notebook pytorch_autograd_and_nn.ipynb will introduce you to the different levels of abstraction that PyTorch provides for building neural network models. You will use this knowledge to implement and train Residual Networks for image classification.

Q2: Image Captioning with Recurrent Neural Networks (40 points)

The notebook rnn_lstm_attention_captioning.ipynb will walk you through the implementation of vanilla recurrent neural networks (RNN) and Long Short Term Memory (LSTM) RNNs. You will use these networks to train an image captioning model. You will then augment your implementation to perform spatial attention over image regions while generating captions.

Q3: Network Visualization (15 points)

The notebook network_visualization.ipynb will walk you through the use of image gradients for generating saliency maps, adversarial examples, and class visualizations.

Q4: Style Transfer (15 points)

In the notebook style_transfer.ipynb, you will learn how to create images with the artistic style of one image and the content of another image.

Steps

1. Click “Open in Colab”

This will launch the corresponding notebook in Google Colab. No installation or setup required!

For more information on using Colab, see our Colab tutorial.

2. Save a copy in Drive

Once the notebook launches, click File -> “Save a copy in Drive…”. This will save a copy of the notebook in your own Google Drive account.

By default, when you save a copy the name will be prepended with “Copy of” – for example saving a copy of the notebook “fully_connected_networks.ipynb” will create a file named “Copy of fully_connected_networks.ipynb”. You should rename your copy to have the same name as the original file.

3. Work on the assignment

Work through the notebook, executing cells and writing code as indicated. You can save your work in Google Drive (click “File” -> “Save”) and resume later if you don’t want to complete it all at once.

While working on the assignment, keep the following in mind:

The notebook has clearly marked blocks where you are expected to write code. Do not write or modify any code outside of these blocks.
Do not add or delete cells from the notebook. You may add new cells to perform scratch computations, but you should delete them before submitting your work.
Run all cells before submitting. You will only get credit for code that has been run.

4. Download .ipynb file

Once you have completed a notebook, download the completed notebook by clicking “File” -> “Download .ipynb”.

Make sure your downloaded file has the same name as the original notebook; either fully_connected_networks.ipynb or convolutional_networks.ipynb for this assignment.

5. Submit your work to Canvas

Create a .zip file containing your completed notebook; name it uniquename_umid.zip (e.g. justincj_12345678.zip).

Make sure you do not change the filenames or include any other files. Your submitted .zip file should contain two files named fully_connected_networks.ipynb and convolutional_networks.ipynb.

We have written a validation script for you to check the structure of your submitted .zip file. This script does not check whether your homework is correct; it only makes sure that your submitted file has the right structure, that you didn’t modify any parts of the .ipynb files that you shouldn’t have, and that you didn’t miss any sections where you were supposed to write code. In order to be graded, your assignment must pass this validation script. It is your responsibility to make sure your assignment is properly formatted before you submit it.

When you are done, upload your work to Canvas (UMich students only).