Task description

Task objective: Annotate text for emotions (e.g. joy, fear, surprise) and/or for polarity orientation (positive/negative).

Task organizers: Carlo Strapparava and Rada Mihalcea

This task is intended as an exploration of the connection between lexical semantics and emotions. All words can potentially convey affective meaning. Every word, even those that are apparently neutral, can evoke pleasant or painful experiences due to their semantic relation with emotional concepts or categories. While some words have emotional meaning with respect to an individual story, for many others the affective power is part of the collective imagination (e.g. words such as "mum", "ghost", "war"). This latter group of words are particularly interesting, because their affective meaning is part of common sense knowledge and can be detected in the linguistic usage. For this reason, we believe it is important to study the use of words in textual productions, and possibly their co-occurrence with words in which the affective meaning is explicit. Several previous studies in linguistics and psychology have considered research issues related to the affective lexicon. For example Ortony et al. [Ortony et al., 1987] distinguishes between words directly referring to emotional states (e.g. "fear", "cheerful") and those having only an indirect reference that depends on the context (e.g. words that indicate possible emotional causes such as "killer" or emotional responses such as "cry").

To explore the connection between emotions and lexical semantics we propose to focus on the emotion classification of news headlines extracted from news web sites. The news headlines typically consist of a few words and are often written by creative people with the intention to "provoke" emotions, and consequently to attract the readers' attention. These characteristics make the news headlines particularly suitable for use in an automatic emotion recognition setting, as the affective/emotional features (if present) are guaranteed to appear in these short sentences.

Datasets and Evaluation

Download the development and test data.

The structure of the task is as follows:

• Corpus: News headlines, extracted from news web sites (such as Google news, CNN) and/or newspapers.
• Objective: Provided a set of predefined emotion labels (e.g. joy, fear, surprise), classify the titles with the appropriate emotion label and with a valence indication (positive/negative).

The emotion annotation and the valence labeling will be regarded as two separate subtasks, and therefore a team can choose to participate in only one or both annotation tasks.

The task will be carried out in an unsupervised setting, and consequently no training will be provided. The reason behind this decision is that we want to emphasize the study of emotion lexical semantics, and avoid biasing the participants toward simple "text categorization" approaches. Nonetheless supervised systems will be not precluded from the participation, and in such cases the participating teams will be allowed to create their own supervised training sets.

Timeline

The task organizers will provide in advance the set of emotion labels and a development corpus. The timeline of the task will follow the general Semeval-2007 timeframe as follows:

• 01/18/2007: a development corpus of 250 headlines annotated for emotions.
• 02/26/2007: a test corpus of 1000 headlines annotated for emotions.
• [deadline]: the participants submit the headlines annotated with emotion labels and/or a valence indication (positive/negative); as done in previous Senseval evaluation, the participating teams are free to choose their own submission deadline, as long as it falls within the window of time imposed by Semeval-2007.

Resources

Participants are free to use any resources they wish. We provide here a set words extracted from WordNet Affect, relevant to the six emotions of interest. Note however that the use of this list of words is entirely optional.

• Emotion words from WordNet Affect [download]

Systems and Results

Systems and results are now described in the Semeval proceedings.

Bibliography

• Sentiment classification bibliography

 For more information, visit the SemEval-2007 home page.