Laboratory of Computer and Information Science / Neural Networks Research Centre CIS Lab Helsinki University of Technology


There are a number of data files involved in this challenge. Each type of file is available for each language.


Word list (input)

First and foremost, there is a list of word forms. The words have been extracted from a text corpus, and each word in the list is preceded by its frequency in the corpus used.

For instance, a subset of the supplied English word list looks like this:

1 barefoot's
2 barefooted
6699 feet
653 flies
2939 flying
1782 foot
64 footprints

Result file (output, i.e., what to submit)

The participants' task is to return a list containing exactly the same words as in the input, with morpheme analyses provided for each word. The list returned shall not contain the word frequency information.

A submission for the above English words may look like this:

barefoot's      BARE FOOT +GEN
barefooted      BARE FOOT +PAST
feet            FOOT +PL
flies           FLY_N +PL, FLY_V +3SG
flying          FLY_V +PCP1
foot            FOOT
footprints      FOOT PRINT +PL

There are a number of things to note about the result file: Each line of the file contains a word (e.g., "feet") separated from its analysis (e.g., "FOOT +PL") by one TAB character. The word needs to look exactly as it does in the input; no capitalization or change of character encoding is allowed. The analysis contains morpheme labels separated using space. The order in which the labels appear does not matter; e.g., "FOOT +PL" is equivalent to "+PL FOOT". The labels are arbitrary: e.g., instead of using "FOOT" you might use "morpheme784" and instead of "+PL" you might use "morpheme2". However, we strongly recommend you to use intuitive labels, when possible, since they make it easier for anyone to get an idea of the quality of the result by looking at it.

If a word has several interpretations, all interpretations should be supplied: e.g., the word "flies" may be the plural form of the noun "fly" (insect) or the third person singular present tense form of the verb "to fly". The alternative analyses must be separated using a comma, as in: "FLY_N +PL, FLY_V +3SG". The existence of alternative analyses makes the task challenging, and we leave it to the participants to decide how much effort they will put into this aspect of the task. In English, for instance, in order to get a perfect score, it would be necessary to distinguish the different functions of the ending "-s" (plural or person ending) as well as the different parts-of-speech of the stem "ly" (noun or verb). As the results will be evaluated against reference analyses (our so-called gold standard), it is worth reading about the guiding principles used when constructing the gold standard.

As far as we understand, you can use any characters in your morpheme labels except whitespace and comma (,). However, we cannot guarantee that the evaluation scripts will work properly, if your labels contain some "strange" characters.

Text corpus for English, Finnish, German and Turkish

The word list (input data) has been constructed by collecting word forms occurring in text corpora. The text corpora have been obtained by combining collections from the Wortschatz collection at the University of Leipzig (Germany), CLEF, and Europarl corpus. The corpus sizes are 18.8 million sentences for English, 6.6 million for Finnish, 9.7 million for German, and 1 million for Turkish. The corpora have been preprocessed for the Morpho Challenge (tokenized, lower-cased, some conversion of character encodings).

If the participants like to do so, they can use the corpora in order to get information about the context in which the different words occur.

We are most grateful to the University of Leipzig for making the Wortschatz resources available to the Challenge, and in particular we thank Stefan Bordag for his kind assistance.

Gold standard morpheme analyses

The desired "correct" analyses for a random sample of circa 1000 words are supplied for each language. These samples can be used for as a training set for a semi-supervised algorithm. Also given is a development set that can be used in order to get a rough estimate of the performance of the participants' morpheme-analyzing algorithm. If independent training and development sets are not needed by the participant, a combined set can be used as a larger training or development set.

The format of the gold standard file is exactly the same as that of the result file to be submitted. That is, each line contains a word and its analysis. The word is separated from the analysis by a TAB character. Morpheme labels in the analysis are separated from each other by a space character. For some words there are multiple correct analyses. These alternative analyses are separated by a comma (,). Examples:

Language Examples
English baby-sitters       baby_N sit_V er_s +PL
indoctrinated      in_p doctrine_N ate_s +PAST
Finnish linuxiin           linux_N +ILL
makaronia          makaroni_N +PTV
German choreographische   choreographie_N isch +ADJ-e
zurueckzubehalten  zurueck_B zu be halt_V +INF
Turkish kontrole           kontrol +DAT
popUlerliGini      popUler +DER_lHg +POS2S +ACC, popUler +DER_lHg +POS3 +ACC3

The English and German gold standards are based on the CELEX data base. The Finnish gold standard is based on the two-level morphology analyzer FINTWOL from Lingsoft, Inc. The Turkish gold-standard analyses have been obtained from a morphological parser developed at Boğaziçi University; it is based on Oflazer's finite-state machines, with a number of changes. We are indebted to Ebru Arısoy for making the Turkish gold standard available to us.

The morphological analyses are morpheme analyses. This means that only grammatical categories that are realized as morphemes are included. For instance, for none of the languages will you find a singular morpheme for nouns or a present-tense morpheme for verbs, because these grammatical categories do not alter or add anything to the word form, in contrast to, e.g., the plural form of a noun (house vs. house+s), or the past tense of verbs (help vs. help+ed, come vs. came).

The morpheme labels that correspond to inflectional (and sometimes also derivational) affixes have been marked with an initial plus sign (e.g., +PL, +PAST). This is due to a feature of the evaluation script: in addition to the overall performance statistics, evaluation measures are also computed separately for the labels starting with a plus sign and those without an initial plus sign. It is thus possible to make an approximate assessment of how accurately affixes are analyzed vs. non-affixes (mostly stems). If you use the same naming convention when labeling the morphemes proposed by your algorithm, this kind of statistics will be available for your output (see the evaluation page for more information).

The morpheme labels that have not been marked as affixes (no initial plus sign) are typically stems. These labels consist of an intuitive string, usually followed by an underscore character (_) and a part-of-speech tag, e.g., "baby_N", "sit_V". In many cases, especially in English, the same morpheme can function as different parts-of-speech; e.g., the English word "force" can be a noun or a verb. In the majority of these cases, however, if there is only a difference in syntax (and not in meaning), the morpheme has been labeled as either a noun or a verb, throughout. For instance, the "original" part-of-speech of "force" is a noun, and consequently both noun and verb inflections of "force" contain the morpheme "force_N":

force      force_N
force's      force_N GEN
forced      force_N +PAST
forces      force_N +3SG, force_N +PL
forcing      force_N +PCP1

Thus, there is not really a need for your algorithm to distinguish between different meanings or syntactic roles of the discovered stem morphemes. However, in some rare cases, if the meanings of the different parts-of-speech do differ clearly, there are two variants, e.g., "train_N" (vehicle), "train_V" (to teach), "fly_N" (insect), "fly_V" (to move through the air). But again, if there are ambiguous meanings within the same part-of-speech, these are not marked in any way, e.g., "fan_N" (device for producing a current of air) vs. "fan_N" (admirer). This notation is a consequence of using CELEX and FINTWOL as the sources for our gold standards. We could have removed the part-of-speech tags, but we decided to leave them there, since they carry useful information without significantly making the task more difficult. There are no part-of-speech tags in the Turkish gold standard.

Random word pairs file

If you want to run evaluation with the development set, you need to download a randomly generated so-called word pairs file for each language to be tested. The word pairs files for the development set contain 300 randomly selected words that are used for estimating recall. For each morpheme of these words, there is another randomly selected word that contains the same morpheme. Read more about this on the evaluation page.

Gold standard segmentations

For some of the languages, we also provide segmentations correspoding to the surface forms of the morpheme labels. The gold standard segmentations should be useful for semi-supervised training of algorithms based solely on segmentation.

The segmentation files contain the segments and their forms combined. The format is similar to that of the result file, but for each morpheme, there are two parts separated by colon (:). The surface form is in the left side of the colon, and the morpheme label is in the right side. If either the surface form or the morpheme does not exist, it is marked by tilde (~). For example, from the English file you can find the following lines:

adversaries     advers:adverse_A ari:ary_s es:+PL
bit     bit:bite_V ~:+PAST, bit:bit_N
where the past tense of "bite" does not have a corresponding segment. If there is a colon in the word form, it is backslashed (\:) in the segment:
mtk:hon	mtk\::mtk hon:+ILL

If you want only the segments, the following UNIX shell command should do the trick:

cat goldstd_trainset.segmentation.eng | sed 's/\([^\]\):[^ ,	]*\( \|,\|$\)/\1\2/g' | \
sed 's/\\:/:/g' | sed 's/~ *//g' | sed 's/ *,/,/g' | sed 's/ *$//' > goldstd_trainset.segments.eng
Note: The whitespace between "," and "]" in the first sed command should be a single TAB character.

While the words are the same as in the files containing only the morpheme labels, please note that the given analyses may be different, due to the morphemes that do not correspond to any segment and the segments that do not correspond to any morpheme.

English and Finnish segmentations are based on the Hutmegs 1.0 package. Ebru Arısoy has provided the Turkish segmentations. Unfortunately, there are no segmentations available for German.

Character encoding

In the source data used for the different languages, there is variation in how accurately certain distinctions are made when letters are rendered. This makes it hard to apply a unified character encoding scheme for all the languages (such as UTF-8). Thus, the following encodings have been used, in which all letters are encoded as one-byte (8-bit) characters:

Standard text. All words are lower-cased, also proper names.
ISO Latin 1 (ISO 8859-1). The Scandinavian special letters å, ä, ö (as well as other letters occuring in loan words, e.g., ü, é, à) are rendered as one-byte characters. All words are lower-cased, also proper names.
Standard text. All words are lower-cased, also all nouns. The German umlaut letters are rendered as the corresponding non-umlaut letter followed by "e", e.g., "laender" (Länder), "koennte" (könnte), "fuer" (für). Double-s is rendered as "ss", e.g., "strasse" (Straße). This coarse encoding is due to the fact that CELEX, the source for the morphological gold standard, utilizes this scheme. Note, however, that in the data you may see special letters encoded using ISO Latin 1 in some loan words, e.g., "société", "l'unità" (these words are not included in CELEX and their analyses will not be evaluated).
Standard text. All words are lower-cased. The letters specific to the Turkish language are replaced by capital letters of the standard Latin alphabet, e.g., "açıkgörüşlülüğünü" is spelled "aCIkgOrUSlUlUGUnU".

Download data

For strictly unsupervised learning, you need only the word lists. For semi-supervised learning, you can use either the gold standard segmentations (for those languages that they are available) or directly the gold standard labels (for any language). Furthermore, there are two independent sets for semi-supervised learning: Training set is meant to be used directly in training, whereas development set is meant to be used for tuning the (possible) parameters of the learning method. If you need only one of them, you can use the combined set instead. In the extended abstract of your submission, you should clearly state which of the data sets are used and to which purpose.

Language Word list Training set (Gold standard labels) Training set (Gold standard segmentations) Development set (Gold standard labels) Development set (word pairs file) Combined Training+Development set (Gold standard labels) Combined Training+Development set (Gold standard segmentations)
English Text Text gzipped Text Text Text Text Text Text
Finnish Text Text gzipped Text Text Text Text Text Text
German Text Text gzipped Text (not available) Text Text Text (not available)
Turkish Text Text gzipped Text Text Text Text Text Text

Or download all the above files as a zip file.


You are at: CIS → Unsupervised Morpheme Analysis -- Morpho Challenge 2010

Page maintained by webmaster at, last updated Tuesday, 29-Jun-2010 09:47:03 EEST