Task #8: Metonymy Resolution at SemEval-2007

Mailing list

We've set up a mailing list to facilitate discussion and information exchange about this task. We strongy encourage you to join it.

After you've signed in you can browse the e-mail discussion on the task.
To join enter your e-mail here:


!! News: Test data available from  the Semeval site. !! Updated and final participation guidelines are part of the test data but can also be downloaded here without a time limit.


Datasets and Formats

Both training/trial and test data are in the same format and both are downloadable from the
the Semeval site:.  Part  A of each Dataset consists of country names annotated for metonymies; Part B consists of company names annotated for metonymies.

Each Part of the trial/training data contains ca. 1000 training instances, where each training instance is a name presented within a four-sentence context (two sentences before and one after the sentence containing the possibly metonymic name) from the British National Corpus. Each name is annotated with a literal reading, a metonymic pattern or an unconventional metonymic reading. We provide the data in the following three formats:

  • a plain text file containing both the text and the metonymy annotation but minimal further mark-up
  • an XML file containing both the text and the metonymy annotation, with further markup, including information about the text type, word and sentence boundaries and POS tags
  • an XML file containing containing the tagged and tokenised text plus standoff metonymy annotation that maps to the text file
Additionally, we provide the following data for help with system development:
  • a separate file containing grammatical relation annotation (in dependency format). This annotation was carried out manually.
  • annotation schemes for metonymy annotation and grammatical relation annotation
  • an evaluation script
  • potentially further scripts

Part A of the test data contains 908 instances of country names and Part B 842 instances of company names.

Please note that all the information provided for the training sets is also provided for the test sets with the exception of the metonymy annotation itself, of course, although the string to be classified is appropriately marked. Thus, we provide  grammatical annotation, tagged files from the BNC etc. For any further details, please refer to the documentation included in the release. Guidelines are also directly available here.

Submission

For submission instructions, please consult the guidelines available with
the test set downloads (which are an updated and final version of the guidelines included in the trial/training set). The updated and final guidelines are also separately available here.

Time Frame

Please note that once you download the test data, you will have maximum of 2  (two) weeks to run your systems and send the results back to us. The trial/training data can be kept as long as you want before submitting your results.

Submission runs

Please note that each registered team is allowed a total of six  runs which are part of a single submission: three for locations (fine, medium, coarse granularity) and three for companies (fine, medium, coarse granularity), in accordance with the requirements of the general Semeval guidelines. All these runs must be incorporated into a single tarball for submission. If multiple tarballs are uploaded all but the last will be automatically discarded.

Evaluation

We encourage systems using any techniques, for example supervised or unsupervised machine learning, knowledge representation or logical inference techniques. We allow partial submissions in case a participant wants to concentrate on specific phenomena only. Participants can choose between several submission formats, mirroring different granularity levels. For example, a system might only distinguish between literal and non-literal readings or it might furnish fine-grained sense distinctions. Systems will be evaluated against the manually annotated unseen test sets, using the following measures. Accuracy is defined as the percentage of correctly classified instances among the samples a system covers. Precision, recall, and balanced f-score will then be used to assess performance with respect to each annotation category. An evaluation script is provided as part of the trial/training and the test data.

Download area

The complete training sets in different formats, submission guidelines, annotation schemes, evaluation software and useful scripts are part of the trial data .

The test sets include again submission guidelines, annotation schemes, evaluation software and useful scripts. Supporting documentation and scripts are the same as those included in the trial/training data, but are part of the test data download for easy access. You can download the test data here

Any further additions will be made available here.

System and Results

This section will be completed after the competition.

References

Sanda Harabagiu (1998). Deriving metonymic coercions from WordNet. In 
Workshop on the Usage of WordNet in Natural Language Processing 
Systems, COLING ACL, 1998, pages 142-148.
S. Kamei and T. Wakao (1992). Metonymy: Reassessment, survey of
acceptability and its treatment in machine translation systems. In
Proc. of ACL, 1992, pages 309-311.
Katja Markert and Udo Hahn (2002). Understanding metonymies in
discourse. Artificial Intelligence, 135(1/2):145-198.
Katja Markert and Malvina Nissim (2002a). Towards a corpus annotated for
metonymies: the case of location names. In Proceedings of the 3rd
International Conference on Language Resources and Evaluation
(LREC2002), pages 1385-1392, Las Palmas, Canary Islands, 2002.
Katja Markert and Malvina Nissim (2002b). Metonymy resolution as a
classification task. In Proceedings of the 2002 Conference on
Empirical Methods in Natural Language Processing, pages 204-213,
Philadelphia, Penn., 6-7 July 2002.
Katja Markert and Malvina Nissim (2006). Metonymic Proper Names: A
Corpus-based Account. In A. Stefanowitsch (ed.), Corpora in Cognitive
Linguistics. Vol. 1: Metaphor and Metonymy, Mouton de Gruyter, 2006.
David Stallard (1993). Two kinds of metonymy. In Proc. of ACL, 1993,
pages 87-94.

 For more information, visit the SemEval-2007 home page.