Emily Mower Provost

Assistant Professor, Computer Science and Engineering

Emily Mower Provost

Assistant Professor, Computer Science and Engineering

My Portrait

Emily Mower Provost

Assistant Professor

University of Michigan
EECS Department
Computer Science & Engineering
3620 BBB
Ann Arbor, MI 48109-2121
Tel: 734-647-1802
Email:



My Portrait

Computational Human-Centered Analysis and Integration

Journal Publications

  1. Yelin Kim and Emily Mower Provost. "ISLA: Temporal Segmentation and Labeling for Audio-Visual Emotion Recognition," IEEE Transactions on Affective Computing, vol: To appear, 2017. [Early Access pdf]

  2. Biqiao (Didi) Zhang, Emily Mower Provost, and Georg Essl. "Cross-corpus Acoustic Emotion Recognition with Multi-task Learning: Seeking Common Ground while Preserving Differences," IEEE Transactions on Affective Computing, vol: To appear, 2017. [pdf]

  3. Duc Le, Keli Licata, Carol Persad, and Emily Mower Provost. "Automatic Assessment of Speech Intelligibility for Individuals with Aphasia," IEEE Transactions on Audio, Speech, and Language Processing, vol: 24, no: 11, Nov. 2016. [pdf]

  4. Carlos Busso, Srinivas Parthasarathy, Alec Burmania, Mohammed AbdelWahab, Najmeh Sadoughi, and Emily Mower Provost, ”MSP-IMPROV: An Acted Corpus of Dyadic Interactions to Study Emotion Perception,” IEEE Transactions on Affective Computing, 8:1(67-80), 2016. [pdf]

  5. Yelin Kim and Emily Mower Provost. “Emotion Recognition During Speech Using Dynamics of Multiple Regions of the Face.” ACM Transactions on Multimedia Computing, Communications and Applications (ACM TOMM), Special Issue on ACM Multimedia Best Papers, 12:1(article 25), 2015. [pdf]

  6. Emily Mower Provost, Yuan (June) Shangguan, Carlos Busso, "UMEME: University of Michigan Emotional McGurk Effect Dataset," IEEE Transactions of Affective Computing, 10:1(395-409), 2015. [pdf]

  7. Chi-Chun Lee, Emily Mower, Carlos Busso, Sungbok Lee and Shrikanth S. Narayanan, Emotion recognition using a hierarchical binary decision tree approach (2011), in: Speech Communication, 53:9-10(1162-1171). [pdf]

  8. Emily Mower, Maja J. Mataric and Shrikanth S. Narayanan, “A Framework for Automatic Human Emotion Classification Using Emotional Profiles,” IEEE Transactions on Audio, Speech and Language Processing, 19:5(1057-1070). May 2011. [pdf]

  9. Emily Mower, Maja Matarić, Shrikanth Narayanan. "Human Perception of Audio-Visual Synthetic Character Emotion Expression in the Presence of Ambiguous and Conflicting Information." IEEE Transactions on Multimedia. 11:5(843-855). August 2009. [pdf]

  10. Carlos Busso, Murtaza Bulut, Chi-Chun Lee, Abe Kazemzadeh, Emily Mower, Samuel Kim, Jeannette Chang, Sungbok Lee, and Shrikanth Narayanan. "IEMOCAP: Interactive emotional dyadic motion capture database." Journal of Language Resources and Evaluation, 42:4(335-359). November 2008. [SpringerLink]

  11. Michael Grimm, Kristian Kroschel, Emily Mower, and Shrikanth Narayanan. "Primitives based estimation and evaluation of emotions in speech." Speech Communication 49, 49:10-11(787-800). November 2007. [pdf]

Conference Publications

  1. Biqiao Zhang, Georg Essl, and Emily Mower Provost. "Predicting the Distribution of Emotion Perception: Capturing Inter-Rater Variability." International Conference on Multimodal Interaction (ICMI). Glasgow, Scotland, November 2017. [Note: full paper, oral presentation] [Coming soon!]

  2. Zakaria Aldeneh, Soheil Khorram, Dimitrios Dimitriadis, Emily Mower Provost. "Pooling Acoustic and Lexical Features for the Prediction of Valence." International Conference on Multimodal Interaction (ICMI). Glasgow, Scotland, November 2017. [Note: short paper, oral presentation] [Coming soon!]

  3. Duc Le, Zakariah Aldeneh, and Emily Mower Provost. “Discretized Continuous Speech Emotion Recognition with Multi-Task Deep Recurrent Neural Network.” Interspeech. Stockholm, Sweden, August 2017. [pdf]

  4. Duc Le, Keli Licata, and Emily Mower Provost. “Automatic Paraphasia Detection from Aphasic Speech: A Preliminary Study.” Interspeech. Stockholm, Sweden, August 2017. [pdf]

  5. John Gideon, Soheil Khorram, Zakariah Aldeneh, Dimitrios Dimitriadis, and Emily Mower Provost. “Progressive Neural Networks for Transfer Learning in Emotion Recognition.” Interspeech. Stockholm, Sweden, August 2017. [pdf]

  6. Soheil Khorram, Zakariah Aldeneh, Dimitrios Dimitriadis, Melvin McInnis, and Emily Mower Provost. “Capturing Long-term Temporal Dependencies with Convolutional Networks for Continuous Emotion.” Interspeech. Stockholm, Sweden, August 2017. [pdf]

  7. Zakariah Aldeneh and Emily Mower Provost. “Using Regional Saliency for Speech Emotion Recognition.” IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP). New Orleans, Louisiana, USA, March 2017. [pdf]

  8. Biqiao Zhang, Georg Essl, and Emily Mower Provost. "Automatic Recognition of Self-Reported and Perceived Emotion: Does Joint Modeling Help?" International Conference on Multimodal Interaction (ICMI). Tokyo, Japan, November 2016. [Note: full paper, oral presentation, best paper honorable mention] [pdf]

  9. Yelin Kim and Emily Mower Provost. "Emotion Spotting: Discovering Regions of Evidence in Audio-Visual Emotion Expressions" International Conference on Multimodal Interaction (ICMI). Tokyo, Japan, November 2016. [Note: full paper, poster presentation] [pdf]

  10. Soheil Khorram, John Gideon, Melvin McInnis, and Emily Mower Provost. "Recognition of Depression in Bipolar Disorder: Leveraging Cohort and Person-Specific Knowledge." Interspeech. San Francisco, CA, September 2016. [pdf]

  11. Duc Le and Emily Mower Provost. "Improving Automatic Recognition of Aphasic Speech with AphasiaBank." Interspeech. San Francisco, CA, September 2016. [pdf]

  12. Biqiao Zhang, Emily Mower Provost, Georg Essl. “Cross-Corpus Acoustic Emotion Recognition From Singing And Speaking: A Multi-Task Learning Approach.” International Conference on Acoustics, Speech and Signal Processing (ICASSP). Shanghai, China, March 2016. [pdf]

  13. John Gideon, Emily Mower Provost, Melvin McInnis. “Mood State Prediction From Speech Of Varying Acoustic Quality For Individuals With Bipolar Disorder.” International Conference on Acoustics, Speech and Signal Processing (ICASSP). Shanghai, China, March 2016. [pdf]

  14. Duc Le and Emily Mower Provost. “Data Selection for Acoustic Emotion Recognition: Analyzing and Comparing Utterance and Sub-Utterance Selection Strategies.” Affective Computing and Intelligent Interaction (ACII). Xi’an, China, September 2015. [Note: oral presentation][pdf]

  15. Biqiao Zhang, Georg Essl, and Emily Mower Provost Georg Essl. “Recognizing Emotion from Singing and Speaking Using Shared Models.” Affective Computing and Intelligent Interaction (ACII). Xi’an, China, September 2015. [Note: oral presentation][pdf]

  16. Yuan (June) Shangguan and Emily Mower Provost. “EmoShapelets: Capturing Local Dynamics of Audiovisual Affective Speech.” Affective Computing and Intelligent Interaction (ACII). Xi’an, China, September 2015. [Note: oral presentation][pdf]

  17. Yelin Kim and Emily Mower Provost. “Leveraging Inter-rater Agreement for Audio-Visual Emotion Recognition. Affective Computing and Intelligent Interaction (ACII). Xi’an, China, September 2015.[pdf]

  18. Yelin Kim, Jixu Chen, Ming-Ching Chang, Xin Wang, Emily Mower Provost, Siwei Lyu. “Modeling Transition Patterns Between Events for Temporal Human Action Segmentation and Classification." IEEE International Conference on Automatic Face and Gesture Recognition (FG). Ljubljana, Slovenia, May, 2015. [Note: oral presentation][pdf]

  19. Biqiao Zhang, Emily Mower Provost, Robert Swedberg, Georg Essl. "Predicting Emotion Perception Across Domains: A Study of Singing and Speaking." AAAI. Austin, TX, USA, January 2015. [pdf]

    Yelin Kim and Emily Mower Provost. "Say Cheese vs. Smile: Reducing
  20. Speech-Related Variability for Facial Emotion Recognition." Proceedings of the ACM International Conference on Multimedia. Florida, USA, November, 2014. [pdf, winner, best student paper!]

  21. Duc Le and Emily Mower Provost. " Modeling Pronunciation, Rhythm, and Intonation for Automatic Assessment of Speech Quality in Aphasia Rehabilitation." Interspeech. Singapore. September, 2014. [pdf]

  22. Duc Le, Keli Licata, Elizabeth Mercado, Carol Persad, Emily Mower Provost. ``Automatic Analysis of Speech Quality for Aphasia Treatment.'' International Conference on Acoustics, Speech and Signal Processing (ICASSP). Florence, Italy. May 2014. [pdf]

  23. Zahi N Karam, Emily Mower Provost, Satinder Singh, Jennifer Montgomery, Christopher Archer, Gloria Harrington, Melvin Mcinnis. ``Ecologically Valid Long-term Mood Monitoring of Individuals with Bipolar Disorder Using Speech.'' International Conference on Acoustics, Speech and Signal Processing (ICASSP). Florence, Italy. May 2014. [pdf]

  24. Duc Le and Emily Mower Provost. “Emotion Recognition From Spontaneous Speech Using Hid- den Markov Models With Deep Belief Networks.” Automatic Speech Recognition and Understanding (ASRU). Olomouc, Czech Republic. December, 2013. [pdf]

  25. Theodora Chaspari, Emily Mower Provost, and Shrikanth S. Narayanan. "Analyzing the Structure of Parent-Moderated Narratives from Children with ASD Using an Entity-Based Approach." Interspeech. Lyon, France. August, 2013. [pdf]

  26. Emily Mower Provost, Irene Zhu, and Shrikanth Narayanan. "Using Emotional Noise to Uncloud Audio-Visual Emotion Perceptual Evaluation." International Conference on Multimedia and Expo (ICME). San Jose, CA. July, 2013. [pdf][Examples][Data Mining Workshop Talk]

  27. Yelin Kim and Emily Mower Provost. "Emotion Classification via Utterance-Level Dynamics: A Pattern-Based Approach to Characterizing Affective Expressions." International Conference on Acoustics, Speech and Signal Processing (ICASSP) . Vancouver, British Columbia, Canada. May, 2013. [pdf]

  28. Yelin Kim, Honglak Lee, and Emily Mower Provost. "Deep Learning for Robust Feature Generation in Audio-Visual Emotion Recognition." International Conference on Acoustics, Speech and Signal Processing (ICASSP) . Vancouver, British Columbia, Canada. May, 2013. [pdf]

  29. Emily Mower Provost. "Identifying Salient Sub-Utterance Emotion Dynamics Using Flexible Units and Estimates of Affective Flow." International Conference on Acoustics, Speech and Signal Processing (ICASSP) . Vancouver, British Columbia, Canada. May, 2013. [pdf]

  30. Emily Mower Provost and Shrikanth Narayanan, "Simplifying Emotion Classification Through Emotion Distillation", Asia-Pacific Signal and Information Processing Association (APSIPA), Los Angeles, CA, 2012. [pdf]

  31. Theodora Chaspari, Emily Mower Provost, Athanasios Katsamanis and Shrikanth Narayanan, "An Acoustic Analysis of Shared Enjoyment in ECA Interactions of Children with Autism", Proceedings of the International Conference on Acoustics, Speech and Signal Processing (ICASSP), Kyoto, Japan, 2012. [pdf]

  32. Emily Mower, Chi-Chun Lee, James Gibson, Theodora Chaspari, Marian Williams, and Shrikanth Narayanan. "Analyzing the Nature of ECA Interactions in Children with Autism." International Speech Communication Association (Interspeech). Florence, Italy, August, 2011. [pdf]

  33. Emily Mower, Matthew Black, Elisa Flores, Marian Williams, and Shrikanth S. Narayanan. "Rachel: Design of an Emotionally Targeted Interactive Agent for Children with Autism." International Conference on Multimedia & Expo (ICME) . Barcelona, Spain, July, 2011. [pdf]

  34. Emily Mower and Shrikanth S. Narayanan. "A Hierarchical Static-Dynamic Framework for Emotion Classification." International Conference on Acoustics, Speech and Signal Processing (ICASSP) . Prague, Czech Republic, May, 2011. [pdf]

  35. Emily Mower, Maja J Matarić, and Shrikanth S. Narayanan. "Robust Representations for Out-of-Domain Emotions Using Emotion Profiles." IEEE Workshop on Spoken Language Technology (SLT). Berkeley, CA, December 2010. [pdf]

  36. Emily Mower, Kyu Jeong Han, Sungbok Lee and Shrikanth S. Narayanan. "A Cluster-Profile Representation of Emotion Using Agglomerative Hierarchical Clustering." International Speech Communication Association (InterSpeech). Makuhari, Japan, September 2010. [pdf]

  37. Dongrui Wu, Thomas Parsons, Emily Mower and Shrikanth S. Narayanan. "Speech Emotion Estimation in 3D Space" IEEE International Conference on Multimedia & Exp (ICME) 2010. Singapore, 2010. [pdf]

  38. Emily Mower, Angeliki Metallinou, Chi-Chun Lee, Abe Kazemzadeh, Carlos Busso, Sungbok Lee, Shrikanth Narayanan. "Interpreting Ambiguous Emotional Expressions." ACII Special Session: Recognition of Non-Prototypical Emotion from Speech- The Final Frontier? (Invited paper). Amsterdam, The Netherlands, September 2009. [pdf]

  39. Emily Mower, Maja J Matarić, Shrikanth Narayanan. "Evaluating Evaluators: A Case Study in Understanding the Benefits and Pitfalls of Multi-Evaluator Modeling." International Speech Communication Association (Interspeech). Brighton, England, September 2009. [pdf]

  40. Chi-Chun Lee, Emily Mower, Carlos Busso, Sungbok Lee and Shrikanth S. Narayanan. "Emotion recognition using a hierarchical binary decision tree approach." International Speech Communication Association (Interspeech). Brighton, England, September 2009. [Emotion Challenge Winner] [pdf]

  41. Emily Mower, Maja J Matarić, Shrikanth Narayanan. "Selection of Emotionally Salient Audio-Visual Features for Modeling Human Evaluations of Synthetic Character Emotion Displays" IEEE International Symposium on Multimedia (ISM) 2008. Berkeley, California, December 2008. [pdf]

  42. Emily Mower, Sungbok Lee, Maja J Matarić, Shrikanth Narayanan. "Joint-processing of audio-visual signals in human perception of conflicting synthetic character emotions." IEEE International Conference on Multimedia & Expo (ICME), Hannover, Germany, June 2008. [pdf]

  43. Emily Mower, Sungbok Lee, Maja J Matarić, Shrikanth Narayanan. "Human perception of synthetic character emotions in the presence of conflicting and congruent vocal and facial expressions." IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP) 2008. Las Vegas, Nevada, March-April 2008. [pdf]

  44. Emily K. Mower, David J. Feil-Seifer, Maja J Matarić, and Shrikanth Narayanan. "Investigating Implicit Cues for User State Estimation in Human-Robot Interaction Using Physiological Measurements." IEEE International Workshop on Robot and Human Interactive Communication (RO-MAN), Jeju Island, South Korea, Aug 2007. [pdf]

  45. Michael Grimm, Emily Mower, Kristian Kroschel, and Shrikanth Narayanan. "Combining categorical and primitives-based emotion recognition." European Signal Processing Conference (EUSIPCO), Florence, Italy, September 2006. [pdf]

  46. Zhou, W., Wu, W., Palmer, N., Mower, E., Daniels, N., Cowen, L., Blumer, A. "Microarray Data Analysis of Survival Times of Patients with Lung Adenocarcinomas Using ADC and K-Medians Clustering." Critical Assessment of Massive Data Analysis (CAMDA). Durham, North Caroline, November 14, 2003. [pdf]