Task #8: Metonymy Resolution at SemEval-2007
set up a mailing list to facilitate discussion and information exchange
about this task. We strongy encourage you to join it.
!! News: Test data available from
the Semeval site.
!! Updated and final
participation guidelines are part of the test data but can also be
without a time limit.
Datasets and Formats
Both training/trial and test data are in the same format and
both are downloadable from the
Semeval site:. Part A
of each Dataset consists of country names annotated for metonymies;
Part B consists
of company names annotated for metonymies.
Each Part of the trial/training data contains
ca. 1000 training instances, where each training instance is a name
presented within a four-sentence context (two sentences before and one
after the sentence containing the possibly metonymic name) from the
British National Corpus. Each name is annotated with a literal
reading, a metonymic pattern or an unconventional metonymic
reading. We provide the data in the following three formats:
Additionally, we provide the following data for help with system
- a plain text file containing both the text and the
metonymy annotation but minimal further mark-up
- an XML file containing both the text and the metonymy
annotation, with further markup, including information about the text
type, word and sentence boundaries and POS tags
- an XML file containing containing the tagged and tokenised
text plus standoff metonymy annotation that maps to the text file
- a separate file containing grammatical relation annotation
(in dependency format). This annotation was carried out manually.
- annotation schemes for metonymy annotation and grammatical
- an evaluation script
- potentially further scripts
Part A of the test data contains 908 instances of country
names and Part B 842 instances of company names.
Please note that all the information provided for the training sets is
also provided for the test sets with the exception of the metonymy
annotation itself, of course, although the string to be classified
is appropriately marked. Thus, we provide grammatical annotation,
tagged files from the BNC etc. For any further details, please refer to
documentation included in the release. Guidelines are also directly
For submission instructions, please consult the
guidelines available with
the test set downloads (which are an updated and final version of the
guidelines included in the trial/training set). The updated and final
guidelines are also separately
Please note that once you download the
test data, you will have maximum of 2 (two) weeks to run
your systems and
send the results back to us. The trial/training data can be kept as
long as you want before submitting your results.
Please note that each registered team
is allowed a total of six runs which are part of a single
submission: three for locations (fine,
medium, coarse granularity) and three for companies (fine, medium,
coarse granularity), in accordance with the requirements of the
general Semeval guidelines.
All these runs must be incorporated into a single tarball for submission. If
multiple tarballs are uploaded all but the last will be automatically
We encourage systems using any techniques, for example
or unsupervised machine learning, knowledge representation or logical
inference techniques. We allow partial submissions in case a
participant wants to concentrate on specific phenomena
only. Participants can choose between several submission formats,
mirroring different granularity levels. For example, a system might
only distinguish between literal and non-literal readings or it might
furnish fine-grained sense distinctions.
Systems will be evaluated against the manually annotated unseen test
using the following measures. Accuracy is defined as the percentage of
correctly classified instances among the samples a system covers.
and balanced f-score will then be used to assess performance with
respect to each annotation category. An evaluation script is provided
as part of the trial/training and the test data.
The complete training sets in different formats, submission
guidelines, annotation schemes, evaluation software and useful scripts
are part of the
trial data .
The test sets include again submission guidelines, annotation
schemes, evaluation software and useful scripts. Supporting
documentation and scripts are the same as those included in the
trial/training data, but are part of the test data download for easy
access. You can download the test data here
Any further additions will be made available here.
System and Results
This section will be completed after the competition.
Sanda Harabagiu (1998). Deriving metonymic coercions from WordNet. In
Workshop on the Usage of WordNet in Natural Language Processing
Systems, COLING ACL, 1998, pages 142-148.
S. Kamei and T. Wakao (1992). Metonymy: Reassessment, survey of
acceptability and its treatment in machine translation systems. In
Proc. of ACL, 1992, pages 309-311.
Katja Markert and Udo Hahn (2002). Understanding metonymies in
discourse. Artificial Intelligence, 135(1/2):145-198.
Katja Markert and Malvina Nissim (2002a). Towards a corpus annotated for
metonymies: the case of location names. In Proceedings of the 3rd
International Conference on Language Resources and Evaluation
(LREC2002), pages 1385-1392, Las Palmas, Canary Islands, 2002.
Katja Markert and Malvina Nissim (2002b). Metonymy resolution as a
classification task. In Proceedings of the 2002 Conference on
Empirical Methods in Natural Language Processing, pages 204-213,
Philadelphia, Penn., 6-7 July 2002.
Katja Markert and Malvina Nissim (2006). Metonymic Proper Names: A
Corpus-based Account. In A. Stefanowitsch (ed.), Corpora in Cognitive
Linguistics. Vol. 1: Metaphor and Metonymy, Mouton de Gruyter, 2006.
David Stallard (1993). Two kinds of metonymy. In Proc. of ACL, 1993,