EECS 598: Special Topics on Natural Language Processing with Deep Learning

Time and Location: Mondays and Wednesdays 12pm - 1:30pm, online via Zoom (Zoom link is provided on Piazza, anyone with umich email can sign up and is welcomed to join the discussions)

Instructor: Lu Wang

Staff and Office Hours:

Prof. Lu Wang, Wednesdays 11am - 12pm, online via Zoom (starting the week of September 2nd, Zoom link is available on Piazza)

Discussion Forum: Piazza, sign up at piazza.com/umich/fall2020/eecs598017

Course Description (and Syllabus)

Deep learning models have made impressive progress in natural language understanding and generation problems, such as automatic text summarization and dialogue systems. In this research-oriented seminar course, we will focus on the discussion of recent advances of using deep learning models for solving natural language generation (NLG) problems. Topics include text summarization, long-form text generation, dialogue systems, question generation, NLG evaluation, and bias and ethics issues in generation systems. The class takes the following format: Students will present and discuss papers, and work on course projects in small groups. After taking this course, students are expected to gain knowledge in major tasks and the state-of-the-arts in NLG, how to design and evaluate NLG systems, how to evaluate research papers in the area, and how to implement and evaluate their own idea.

Note: Students who have not taken any NLP course before and would like to learn fundamental concepts and techniques in NLP should consider EECS 595 (graduate-level NLP, offered every fall or winter semester) or EECS 495 (undergraduate-level NLP, scheduled in winter 2021).

Please find the syllabus here: [Link]

References

This course does not have any required textbook, but you can refer to the following resources.

Jacob Eisenstein, "Introduction to Natural Language Processing", The MIT Press, 2019
Dan Jurafsky and James H. Martin, "Speech and Language Processing, 2nd Edition", Prentice Hall, 2009. Third edition draft is available at web.stanford.edu/~jurafsky/slp3/.

Prerequisites

This course is designed for graduate students and senior undergraduate students majoring computer science, computational linguistics, and other relate areas. Concretely,

Programming: Students are expected to be proficient in some programming languages (Python is encouraged).
Natural language processing/Machine learning knowledge: Students should have familiarity with natural language processing concepts and machine learning fundamentals, e.g., have done projects with machine learning tools to train and evaluate computational and statistical models.

Grading

Each assignment or report is due by the end of day on the corresponding due date (i.e. 11:59pm, EST). Canvas is used for electronic submission. Assignment or report turned in late will be charged 20 points (out of 100 points) off for each late day (i.e. every 24 hours). Each student has a budget of 5 days throughout the semester before a late penalty is applied. You may want to use it wisely, e.g. save for emergencies. Each 24 hours or part thereof that a submission is late uses up one full late day. Late days are not applicable to final presentation. Each group member is charged with the same number of late days, if any, for their submission. There is no need to inform the instructors if late days are used; timestamp of the last submission on Canvas will be used for automatic grade calculation.

Grades will be determined based on paper critiques and presentation, project, and participation:

Paper critiques (38%): write a short (half a page to one page) critique (using this [template]) for each paper for discussion, each submission worth 2%; the critiques are due the day before the class (e.g. for papers discussed on Sep 2, critiques are due on Sep 1)
Paper presentation and discussion leading (10%): each student will present papers and lead corresponding discussions twice (with another student or individually); send your draft slides two days before the class (e.g. if you present on Sep 21, please email the instructor by Sep 19 to get feedback)
Project (40%): team of 2 to 3 students, proposal (5%), reports (10%+15%, mid-tern and final), presentations (5%+5%, mid-term and final)
Project feedback (6%): each student will write short feedback to other groups' projects after mid-term presentation
Participation (6%): attendance, participating in-class discussions, etc

Sample Projects

Here are some sample ideas that are related to the topics discussed in this course, you should feel free to explore other research directions based on your own interests.

Abstractive summarization for news articles (supervised or unsupervised)
Knowledge-driven controlled generation for response generation
Reference-based or reference-free summary evaluation
Bias detection or correction in generation systems
More sample projects from Stanford NLP course [link]

Schedule (tentative)

Aug 31 (introduction)

course introduction, basics of neural models in NLP [slides]
Reading 1: Sequence to Sequence Learning with Neural Networks, Ilya Sutskever, Oriol Vinyals, Quoc V. Le. NeurIPS 2014. [link]
Reading 2: Neural Machine Translation by Jointly Learning to Align and Translate. Dzmitry Bahdanau, Kyunghyun Cho, Yoshua Bengio. ICLR 2015. [link]
TODO: start thinking about projects and looking for teammates

Theme: Text Summarization

Sep 2 (neural abstractive summarization)

Presenter/discussion leader: Lu Wang [slides]
Paper for discussion: Get To The Point: Summarization with Pointer-Generator Networks. Abigail See, Peter J. Liu, Christopher D. Manning. ACL 2017. [link]

Sep 7 (no class, labor day)

Sep 9 (reinforcement learning for summarization)

Presenter/discussion leaders: Lu Wang
Paper for discussion: A deep reinforced model for abstractive summarization. Romain Paulus, Caiming Xiong, and Richard Socher. ICLR 2018. [link]
Reading 1: Fast abstractive summarization with reinforce-selected sentence rewriting. Yen-Chun Chen and Mohit Bansal. ACL 2018. [link]
Reading 2: Knowledge Graph-Augmented Abstractive Summarization with Semantic-Driven Cloze Reward. Luyang Huang, Lingfei Wu, and Lu Wang. ACL 2020. [link]

Sep 14 (project discussion with instructor)

See arrangement on piazza to make appointment with the instructor

Sep 16 (no class and office hour)

Project proposal is due on Sep 18, 11:59pm.

Sep 21 (unsupervised abstractive summarization)

Presenter/discussion leaders: Minxue Niu
Paper for discussion: The Summary Loop: Learning to Write Abstractive Summaries Without Examples. Philippe Laban, Andrew Hsi, John Canny, Marti A. Hearst. ACL 2020. [link]
Reading: Sentence Centrality Revisited for Unsupervised Summarization. Hao Zheng and Mirella Lapata. ACL 2019. [link]

Sep 23 (errors in summaries and evaluation)

Presenter/discussion leaders: Yashmeet Gambhir
Paper for discussion: On faithfulness and factuality in abstractive summarization. Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan McDonald. ACL 2020. [link]
Reading: Question answering as an automatic evaluation metric for news article summarization. Matan Eyal, Tal Baumel, and Michael Elhadad. NAACL 2019. [link]

Theme: Text Generation

Sep 28 (large language models)

Presenter/discussion leaders: Spencer Vagg
Paper for discussion: Language models are unsupervised multitask learners. Radford, Alec, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. OpenAI Blog. 2019. [link]
Reading: BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. ACL 2020. [link]

Sep 30 (knowledge-enhanced generation)

Presenter/discussion leaders: Barrett Lattimer
Paper for discussion: Text Generation from Knowledge Graphs with Graph Transformers. Rik Koncel-Kedziorski, Dhanush Bekal, Yi Luan, Mirella Lapata, Hannaneh Hajishirzi. NAACL 2019. [link]
Reading: Enhancing Topic-to-Essay Generation with External Commonsense Knowledge. Pengcheng Yang, Lei Li, Fuli Luo, Tianyu Liu, Xu Sun. ACL 2019. [link]

Oct 5 (content planning for long-text generation)

Presenter/discussion leaders: Yujian Liu
Paper for discussion: Sentence-Level Content Planning and Style Specification for Neural Text Generation. Xinyu Hua, Lu Wang. EMNL 2019. [link]
Reading: Data-to-Text Generation with Content Selection and Planning. Ratish Puduppully, Li Dong, Mirella Lapata. AAAI 2019. [link]

Oct 7 (non-monotonic decoding)

Presenter/discussion leaders: Bohan Zhang
Paper for discussion: Mask-Predict: Parallel Decoding of Conditional Masked Language Models. Marjan Ghazvininejad, Omer Levy, Yinhan Liu, Luke Zettlemoyer. EMNLP 2019. [link]
Reading: Insertion-based decoding with automatically inferred generation order. Jiatao Gu, Qi Liu, and Kyunghyun Cho. Transactions of the Association for Computational Linguistics (2019). [link]

Oct 12 (issues in neural generation)

Presenter/discussion leaders: Zhengyuan Cui
Paper for discussion: The Curious Case of Neural Text Degeneration. Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, Yejin Choi. ICLR 2020. [link]
Reading: What makes a good conversation? How controllable attributes affect human judgments. Abigail See, Stephen Roller, Douwe Kiela, Jason Weston. NAACL 2019. [link]

Topic: Project Mid-term Presentation

Oct 14

Group presentation
Project progress discussion with instructor, see arrangement on piazza

Oct 19

Group presentation
Project progress discussion with instructor, see arrangement on piazza
Project progress report is due on Oct 20, 11:59pm.

Theme: Dialogue Generation

Oct 21 (goal-oriented dialogues with knowledge augmentation)

Presenter/discussion leaders: Do June Min
Paper for discussion: Wizard of Wikipedia: Knowledge-Powered Conversational agents. Emily Dinan, Stephen Roller, Kurt Shuster, Angela Fan, Michael Auli, Jason Weston. ICLR 2019. [link]
Reading: OpenDialKG: Explainable Conversational Reasoning with Attention-based Walks over Knowledge Graphs. Seungwhan Moon, Pararth Shah, Anuj Kumar, Rajen Subba. ACL 2019. [link]

Oct 26 (personalized dialogue agents)

Presenter/discussion leaders: Xueming Xu
Paper for discussion: Personalizing Dialogue Agents: I have a dog, do you have pets too? Saizheng Zhang, Emily Dinan, Jack Urbanek, Arthur Szlam, Douwe Kiela, Jason Weston. 2018. [link]
Reading: Learning from Dialogue after Deployment: Feed Yourself, Chatbot! Braden Hancock, Antoine Bordes, Pierre-Emmanuel Mazare, Jason Weston. ACL 2019. [link]

Oct 28 (chitchat)

Presenter/discussion leaders: Shangquan Sun
Paper for discussion: Recipes for building an open-domain chatbot. Stephen Roller, Emily Dinan, Naman Goyal, Da Ju, Mary Williamson, Yinhan Liu, Jing Xu, Myle Ott, Kurt Shuster, Eric M. Smith, Y-Lan Boureau, Jason Weston. 2020. [link]
Reading: Towards a Human-like Open-Domain Chatbot. Daniel Adiwardana, Minh-Thang Luong, David R. So, Jamie Hall, Noah Fiedel, Romal Thoppilan, Zi Yang, Apoorv Kulshreshtha, Gaurav Nemade, Yifeng Lu, Quoc V. Le. 2020. [link]

Theme: Question Generation

Nov 2 (factoid question generation)

Presenter/discussion leaders: Jian Zhu
Paper for discussion: Harvesting Paragraph-Level Question-Answer Pairs from Wikipedia. Xinya Du, Claire Cardie. ACL 2018. [link]
Reading: Addressing semantic drift in question generation for semi-supervised question answering. Shiyue Zhang and Mohit Bansal. EMNLP 2019. [link]

Nov 4 (open-ended/complex question generation)

Presenter/discussion leaders: Minxue Niu, Xueming Xu
Paper for discussion: Learning to Ask Good Questions: Ranking Clarification Questions using Neural Expected Value of Perfect Information. Sudha Rao, Hal Daume III. ACL 2018. [link]
Reading: Semantic Graphs for Generating Deep Questions. Liangming Pan, Yuxi Xie, Yansong Feng, Tat-Seng Chua, Min-Yen Kan. ACL 2020. [link]

Theme: NLG Evaluation

Nov 9 (issues in automatic metrics)

Presenter/discussion leaders: Yashmeet Gambhir, Barrett Lattimer
Paper for discussion: Why We Need New Evaluation Metrics for NLG. Jekaterina Novikova, Ondrej Dusek, Amanda Cercas Curry, Verena Rieser. EMNLP 2017. [link]
Reading: Unifying Human and Statistical Evaluation for Natural Language Generation. Tatsunori Hashimoto, Hugh Zhang, Percy Liang. NAACL 2019. [link]

Nov 11 (reference-based evaluation)

Presenter/discussion leaders: Spencer Vagg, Do June Min
Paper for discussion: BLEURT: Learning Robust Metrics for Text Generation. Thibault Sellam, Dipanjan Das, Ankur P. Parikh. ACL 2020. [link]
Reading: BERTScore: Evaluating Text Generation with BERT. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, Yoav Artzi. ICLR 2020. [link]

Nov 16 (reference-free evaluation)

Presenter/discussion leaders: Yujian Liu, Bohan Zhang
Paper for discussion: How NOT To Evaluate Your Dialogue System: An Empirical Study of Unsupervised Evaluation Metrics for Dialogue Response Generation. Chia-Wei Liu, Ryan Lowe, Iulian Serban, Mike Noseworthy, Laurent Charlin, Joelle Pineau. EMNLP 2016. [link]
Reading: Unsupervised Quality Estimation for Neural Machine Translation. Marina Fomicheva, Shuo Sun, Lisa Yankovskaya, Frederic Blain, Francisco Guzman, Mark Fishel, Nikolaos Aletras, Vishrav Chaudhary, Lucia Specia. TACL 2020. [link]

Theme: Bias and Ethics for Neural Generation

Nov 18 (can human tell machine-generated text?)

Presenter/discussion leaders: Zhengyuan Cui
Paper for discussion: Defending Against Neural Fake News. Rowan Zellers, Ari Holtzman, Hannah Rashkin, Yonatan Bisk, Ali Farhadi, Franziska Roesner, Yejin Choi. NeurIPS 2019. [link]
Reading: Automatic Detection of Generated Text is Easiest when Humans are Fooled. Daphne Ippolito, Daniel Duckworth, Chris Callison-Burch, Douglas Eck. ACL 2020. [link]

Nov 30 (ethics in neural generation)

Presenter/discussion leaders: Shangquan Sun, Jian Zhu
Paper for discussion: Universal Adversarial Triggers for Attacking and Analyzing NLP. Eric Wallace, Shi Feng, Nikhil Kandpal, Matt Gardner, Sameer Singh. EMNLP 2019. [link]
Reading: The Woman Worked as a Babysitter: On Biases in Language Generation. Emily Sheng, Kai-Wei Chang, Premkumar Natarajan, Nanyun Peng. EMNLP 2019. [link]