Word and phrase coding

Note

NLTK provides documentation for each tag, which can be queried using
the tag, e.g. nltk.help.upenn_tagset(‘RB’), or a regular
expression, e.g. nltk.help.upenn_tagset(‘NN.*’).
Some corpora have README files with tagset documentation,
see nltk.corpus.???.readme(), substituting in the name
of the corpus.

Note

Your Turn:
Many words, like ski and race, can be used as nouns
or verbs with no difference in pronunciation. Can you think of
others? Hint: think of a commonplace object and try to put
the word to before it to see if it can also be a verb, or
think of an action and try to put the before it to see if
it can also be a noun. Now make up a sentence with both uses
of this word, and run the POS-tagger on this sentence.

2.1   Representing Tagged Tokens

By convention in NLTK, a tagged token is represented using a
tuple consisting of the token and the tag.
We can create one of these special tuples from the standard string
representation of a tagged token, using the function str2tuple():

>>> tagged_token = nltk.tag.str2tuple('fly/NN')
>>> tagged_token
('fly', 'NN')
>>> tagged_token[0]
'fly'
>>> tagged_token[1]
'NN'

We can construct a list of tagged tokens directly from a string. The first
step is to tokenize the string
to access the individual word/tag strings, and then to convert
each of these into a tuple (using str2tuple()).

>>> sent = '''
... The/AT grand/JJ jury/NN commented/VBD on/IN a/AT number/NN of/IN
... other/AP topics/NNS ,/, AMONG/IN them/PPO the/AT Atlanta/NP and/CC
... Fulton/NP-tl County/NN-tl purchasing/VBG departments/NNS which/WDT it/PPS
... said/VBD ``/`` ARE/BER well/QL operated/VBN and/CC follow/VB generally/RB
... accepted/VBN practices/NNS which/WDT inure/VB to/IN the/AT best/JJT
... interest/NN of/IN both/ABX governments/NNS ''/'' ./.
... '''
>>> [nltk.tag.str2tuple(t) for t in sent.split()]
[('The', 'AT'), ('grand', 'JJ'), ('jury', 'NN'), ('commented', 'VBD'),
('on', 'IN'), ('a', 'AT'), ('number', 'NN'), ... ('.', '.')]

2.2   Reading Tagged Corpora

Several of the corpora included with NLTK have been tagged for
their part-of-speech. Here’s an example of what you might see if you
opened a file from the Brown Corpus with a text editor:

The/at Fulton/np-tl County/nn-tl Grand/jj-tl Jury/nn-tl
said/vbd Friday/nr an/at investigation/nn of/in Atlanta’s/np$
recent/jj primary/nn election/nn produced/vbd / no/at
evidence/nn »/» that/cs any/dti irregularities/nns took/vbd
place/nn ./.

Other corpora use a variety of formats for storing part-of-speech tags.
NLTK’s corpus readers provide a uniform interface so that you
don’t have to be concerned with the different file formats.
In contrast with the file fragment shown above,
the corpus reader for the Brown Corpus represents the data as shown below.
Note that part-of-speech tags have been converted to uppercase, since this has
become standard practice since the Brown Corpus was published.

>>> nltk.corpus.brown.tagged_words()
[('The', 'AT'), ('Fulton', 'NP-TL'), ...]
>>> nltk.corpus.brown.tagged_words(tagset='universal')
[('The', 'DET'), ('Fulton', 'NOUN'), ...]

Whenever a corpus contains tagged text, the NLTK corpus interface
will have a tagged_words() method.
Here are some more examples, again using the output format
illustrated for the Brown Corpus:

>>> print(nltk.corpus.nps_chat.tagged_words())
[('now', 'RB'), ('im', 'PRP'), ('left', 'VBD'), ...]
>>> nltk.corpus.conll2000.tagged_words()
[('Confidence', 'NN'), ('in', 'IN'), ('the', 'DT'), ...]
>>> nltk.corpus.treebank.tagged_words()
[('Pierre', 'NNP'), ('Vinken', 'NNP'), (',', ','), ...]

Not all corpora employ the same set of tags; see the
tagset help functionality and the readme() methods
mentioned above for documentation.
Initially we want to avoid the complications of these tagsets,
so we use a built-in mapping to the «Universal Tagset»:

>>> nltk.corpus.brown.tagged_words(tagset='universal')
[('The', 'DET'), ('Fulton', 'NOUN'), ...]
>>> nltk.corpus.treebank.tagged_words(tagset='universal')
[('Pierre', 'NOUN'), ('Vinken', 'NOUN'), (',', '.'), ...]

Tagged corpora for several other languages are distributed with NLTK,
including Chinese, Hindi, Portuguese, Spanish, Dutch and Catalan.
These usually contain non-ASCII text,
and Python always displays this in hexadecimal when printing a larger structure
such as a list.

>>> nltk.corpus.sinica_treebank.tagged_words()
[('ä', 'Neu'), ('åæ', 'Nad'), ('åç', 'Nba'), ...]
>>> nltk.corpus.indian.tagged_words()
[('মহিষের', 'NN'), ('সন্তান', 'NN'), (':', 'SYM'), ...]
>>> nltk.corpus.mac_morpho.tagged_words()
[('Jersei', 'N'), ('atinge', 'V'), ('mxe9dia', 'N'), ...]
>>> nltk.corpus.conll2002.tagged_words()
[('Sao', 'NC'), ('Paulo', 'VMI'), ('(', 'Fpa'), ...]
>>> nltk.corpus.cess_cat.tagged_words()
[('El', 'da0ms0'), ('Tribunal_Suprem', 'np0000o'), ...]

If your environment is set up correctly, with appropriate editors and fonts,
you should be able to display individual strings in a human-readable way.
For example, 2.1 shows data accessed using
nltk.corpus.indian.

If the corpus is also segmented into sentences, it will have
a tagged_sents() method that divides up the tagged words into
sentences rather than presenting them as one big list.
This will be useful when we come to developing automatic taggers,
as they are trained and tested on lists of sentences, not words.

2.4   Nouns

Nouns generally refer to people, places, things, or concepts, e.g.:
woman, Scotland, book, intelligence. Nouns can appear after
determiners and adjectives, and can be the subject or object of the
verb, as shown in 2.2.

Table 2.2:

Syntactic Patterns involving some Nouns

The simplified noun tags are N for common nouns like book,
and NP for proper nouns like Scotland.

Let’s inspect some tagged text to see what parts of speech occur before a noun,
with the most frequent ones first. To begin with, we construct a list
of bigrams whose members are themselves word-tag pairs such as
((‘The’, ‘DET’), (‘Fulton’, ‘NP’)) and ((‘Fulton’, ‘NP’), (‘County’, ‘N’)).
Then we construct a FreqDist from the tag parts of the bigrams.

>>> word_tag_pairs = nltk.bigrams(brown_news_tagged)
>>> noun_preceders = [a[1] for (a, b) in word_tag_pairs if b[1] == 'NOUN']
>>> fdist = nltk.FreqDist(noun_preceders)
>>> [tag for (tag, _) in fdist.most_common()]
['NOUN', 'DET', 'ADJ', 'ADP', '.', 'VERB', 'CONJ', 'NUM', 'ADV', 'PRT', 'PRON', 'X']

This confirms our assertion that nouns occur after determiners and
adjectives, including numeral adjectives (tagged as NUM).

2.5   Verbs

Verbs are words that describe events and actions, e.g. fall,
eat in 2.3.
In the context of a sentence, verbs typically express a relation
involving the referents of one or more noun phrases.

Table 2.3:

Syntactic Patterns involving some Verbs

What are the most common verbs in news text? Let’s sort all the verbs by frequency:

>>> wsj = nltk.corpus.treebank.tagged_words(tagset='universal')
>>> word_tag_fd = nltk.FreqDist(wsj)
>>> [wt[0] for (wt, _) in word_tag_fd.most_common() if wt[1] == 'VERB']
['is', 'said', 'are', 'was', 'be', 'has', 'have', 'will', 'says', 'would',
 'were', 'had', 'been', 'could', "'s", 'can', 'do', 'say', 'make', 'may',
 'did', 'rose', 'made', 'does', 'expected', 'buy', 'take', 'get', 'might',
 'sell', 'added', 'sold', 'help', 'including', 'should', 'reported', ...]

Note that the items being counted in the frequency distribution are word-tag pairs.
Since words and tags are paired, we can treat the word as a condition and the tag
as an event, and initialize a conditional frequency distribution with a list of
condition-event pairs. This lets us see a frequency-ordered list of tags given a word:

>>> cfd1 = nltk.ConditionalFreqDist(wsj)
>>> cfd1['yield'].most_common()
[('VERB', 28), ('NOUN', 20)]
>>> cfd1['cut'].most_common()
[('VERB', 25), ('NOUN', 3)]

We can reverse the order of the pairs, so that the tags are the conditions, and the
words are the events. Now we can see likely words for a given tag. We
will do this for the WSJ tagset rather than the universal tagset:

>>> wsj = nltk.corpus.treebank.tagged_words()
>>> cfd2 = nltk.ConditionalFreqDist((tag, word) for (word, tag) in wsj)
>>> list(cfd2['VBN'])
['been', 'expected', 'made', 'compared', 'based', 'priced', 'used', 'sold',
'named', 'designed', 'held', 'fined', 'taken', 'paid', 'traded', 'said', ...]

To clarify the distinction between VBD (past tense) and VBN
(past participle), let’s find words which can be both VBD and
VBN, and see some surrounding text:

>>> [w for w in cfd1.conditions() if 'VBD' in cfd1[w] and 'VBN' in cfd1[w]]
['Asked', 'accelerated', 'accepted', 'accused', 'acquired', 'added', 'adopted', ...]
>>> idx1 = wsj.index(('kicked', 'VBD'))
>>> wsj[idx1-4:idx1+1]
[('While', 'IN'), ('program', 'NN'), ('trades', 'NNS'), ('swiftly', 'RB'),
 ('kicked', 'VBD')]
>>> idx2 = wsj.index(('kicked', 'VBN'))
>>> wsj[idx2-4:idx2+1]
[('head', 'NN'), ('of', 'IN'), ('state', 'NN'), ('has', 'VBZ'), ('kicked', 'VBN')]

In this case, we see that the past participle of kicked is preceded by a form of
the auxiliary verb have. Is this generally true?

2.6   Adjectives and Adverbs

Two other important word classes are adjectives and adverbs.
Adjectives describe nouns, and can be used as modifiers
(e.g. large in the large pizza), or in predicates (e.g. the
pizza is large). English adjectives can have internal structure
(e.g. fall+ing in the falling
stocks). Adverbs modify verbs to specify the time, manner, place or
direction of the event described by the verb (e.g. quickly in
the stocks fell quickly). Adverbs may also modify adjectives
(e.g. really in Mary’s teacher was really nice).

English has several categories of closed class words in addition to
prepositions, such as articles (also often called determiners)
(e.g., the, a), modals (e.g., should,
may), and personal pronouns (e.g., she, they).
Each dictionary and grammar classifies these words differently.

2.8   Exploring Tagged Corpora

Let’s briefly return to the kinds of exploration of corpora we saw in previous chapters,
this time exploiting POS tags.

Suppose we’re studying the word often and want to see how it is used
in text. We could ask to see the words that follow often

>>> brown_learned_text = brown.words(categories='learned')
>>> sorted(set(b for (a, b) in nltk.bigrams(brown_learned_text) if a == 'often'))
[',', '.', 'accomplished', 'analytically', 'appear', 'apt', 'associated', 'assuming',
'became', 'become', 'been', 'began', 'call', 'called', 'carefully', 'chose', ...]

However, it’s probably more instructive to use the tagged_words()
method to look at the part-of-speech tag of the following words:

>>> brown_lrnd_tagged = brown.tagged_words(categories='learned', tagset='universal')
>>> tags = [b[1] for (a, b) in nltk.bigrams(brown_lrnd_tagged) if a[0] == 'often']
>>> fd = nltk.FreqDist(tags)
>>> fd.tabulate()
 PRT  ADV  ADP    . VERB  ADJ
   2    8    7    4   37    6

Notice that the most high-frequency parts of speech following often are verbs.
Nouns never appear in this position (in this particular corpus).

Next, let’s look at some larger context, and find words involving
particular sequences of tags and words (in this case «<Verb> to <Verb>»).
In code-three-word-phrase we consider each three-word window in the sentence ,
and check if they meet our criterion . If the tags
match, we print the corresponding words .

Finally, let’s look for words that are highly ambiguous as to their part of speech tag.
Understanding why such words are tagged as they are in each context can help us clarify
the distinctions between the tags.

>>> brown_news_tagged = brown.tagged_words(categories='news', tagset='universal')
>>> data = nltk.ConditionalFreqDist((word.lower(), tag)
...                                 for (word, tag) in brown_news_tagged)
>>> for word in sorted(data.conditions()):
...     if len(data[word]) > 3:
...         tags = [tag for (tag, _) in data[word].most_common()]
...         print(word, ' '.join(tags))
...
best ADJ ADV NP V
better ADJ ADV V DET
close ADV ADJ V N
cut V N VN VD
even ADV DET ADJ V
grant NP N V -
hit V VD VN N
lay ADJ V NP VD
left VD ADJ N VN
like CNJ V ADJ P -
near P ADV ADJ DET
open ADJ V N ADV
past N ADJ DET P
present ADJ ADV V N
read V VN VD NP
right ADJ N DET ADV
second NUM ADV DET N
set VN V VD N -
that CNJ V WH DET

3.1   Indexing Lists vs Dictionaries

A text, as we have seen, is treated in Python as a list of words.
An important property of lists is that we can «look up» a particular
item by giving its index, e.g. text1[100]. Notice how we specify
a number, and get back a word. We can think of a list as a simple
kind of table, as shown in 3.1.

Contrast this situation with frequency distributions (3),
where we specify a word, and get back a number, e.g. fdist[‘monstrous’], which
tells us the number of times a given word has occurred in a text. Look-up using words is
familiar to anyone who has used a dictionary. Some more examples are shown in
3.2.

In the case of a phonebook, we look up an entry using a name,
and get back a number. When we type a domain name in a web browser,
the computer looks this up to get back an IP address. A word
frequency table allows us to look up a word and find its frequency in
a text collection. In all these cases, we are mapping from names to
numbers, rather than the other way around as with a list.
In general, we would like to be able to map between
arbitrary types of information. 3.1 lists a variety
of linguistic objects, along with what they map.

Table 3.1:

Linguistic Objects as Mappings from Keys to Values

Most often, we are mapping from a «word» to some structured object.
For example, a document index maps from a word (which we can represent
as a string), to a list of pages (represented as a list of integers).
In this section, we will see how to represent such mappings in Python.

3.2   Dictionaries in Python

Python provides a dictionary data type that can be used for
mapping between arbitrary types. It is like a conventional dictionary,
in that it gives you an efficient way to look things up. However,
as we see from 3.1, it has a much wider range of uses.

To illustrate, we define pos to be an empty dictionary and then add four
entries to it, specifying the part-of-speech of some words. We add
entries to a dictionary using the familiar square bracket notation:

So, for example, says that
the part-of-speech of colorless is adjective, or more
specifically, that the key ‘colorless’
is assigned the value ‘ADJ’ in dictionary pos.
When we inspect the value of pos we see
a set of key-value pairs. Once we have populated the dictionary
in this way, we can employ the keys to retrieve values:

>>> pos['ideas']
'N'
>>> pos['colorless']
'ADJ'

Of course, we might accidentally use a key that hasn’t been assigned a value.

>>> pos['green']
Traceback (most recent call last):
  File "<stdin>", line 1, in ?
KeyError: 'green'

This raises an important question. Unlike lists and strings, where we
can use len() to work out which integers will be legal indexes,
how do we work out the legal keys for a dictionary? If the dictionary
is not too big, we can simply inspect its contents by evaluating the
variable pos. As we saw above (line ), this gives
us the key-value pairs. Notice that they are not in the same order they
were originally entered; this is because dictionaries are not sequences
but mappings (cf. 3.2), and the keys are not inherently
ordered.

Alternatively, to just find the keys, we can convert the
dictionary to a list — or use
the dictionary in a context where a list is expected,
as the parameter of sorted() ,
or in a for loop .

As well as iterating over all keys
in the dictionary with a for loop, we can use the for loop
as we did for printing lists:

>>> for word in sorted(pos):
...     print(word + ":", pos[word])
...
colorless: ADJ
furiously: ADV
sleep: V
ideas: N

Finally, the dictionary methods keys(), values() and
items() allow us to access the keys, values, and key-value pairs as separate lists.
We can even sort tuples , which orders them according to their first element
(and if the first elements are the same, it uses their second elements).

>>> list(pos.keys())
['colorless', 'furiously', 'sleep', 'ideas']
>>> list(pos.values())
['ADJ', 'ADV', 'V', 'N']
>>> list(pos.items())
[('colorless', 'ADJ'), ('furiously', 'ADV'), ('sleep', 'V'), ('ideas', 'N')]
>>> for key, val in sorted(pos.items()): 
...     print(key + ":", val)
...
colorless: ADJ
furiously: ADV
ideas: N
sleep: V

We want to be sure that when we look something up in a dictionary, we
only get one value for each key. Now
suppose we try to use a dictionary to store the fact that the
word sleep can be used as both a verb and a noun:

>>> pos['sleep'] = 'V'
>>> pos['sleep']
'V'
>>> pos['sleep'] = 'N'
>>> pos['sleep']
'N'

Initially, pos[‘sleep’] is given the value ‘V’. But this is
immediately overwritten with the new value ‘N’.
In other words, there can only be one entry in the dictionary for ‘sleep’.
However, there is a way of storing multiple values in
that entry: we use a list value,
e.g. pos[‘sleep’] = [‘N’, ‘V’]. In fact, this is what we
saw in 4 for the CMU Pronouncing Dictionary,
which stores multiple pronunciations for a single word.

3.3   Defining Dictionaries

We can use the same key-value pair format to create a dictionary. There’s
a couple of ways to do this, and we will normally use the first:

>>> pos = {'colorless': 'ADJ', 'ideas': 'N', 'sleep': 'V', 'furiously': 'ADV'}
>>> pos = dict(colorless='ADJ', ideas='N', sleep='V', furiously='ADV')

Note that dictionary keys must be immutable types, such as strings and tuples.
If we try to define a dictionary using a mutable key, we get a TypeError:

>>> pos = {['ideas', 'blogs', 'adventures']: 'N'}
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: list objects are unhashable

3.4   Default Dictionaries

If we try to access a key that is not in a dictionary, we get an error.
However, its often useful if a dictionary can automatically create
an entry for this new key and give it a default value, such as zero or
the empty list. For this reason, a special kind of dictionary
called a defaultdict is available.
In order to use it, we have to supply a parameter which can be used to
create the default value, e.g. int, float, str, list, dict,
tuple.

>>> from collections import defaultdict
>>> frequency = defaultdict(int)
>>> frequency['colorless'] = 4
>>> frequency['ideas']
0
>>> pos = defaultdict(list)
>>> pos['sleep'] = ['NOUN', 'VERB']
>>> pos['ideas']
[]

The above examples specified the default value of a dictionary entry to
be the default value of a particular data type. However, we can specify
any default value we like, simply by providing the name of a function
that can be called with no arguments to create the required value.
Let’s return to our part-of-speech example, and create a dictionary
whose default value for any entry is ‘N’ .
When we access a non-existent entry ,
it is automatically added to the dictionary .

>>> f = lambda: 'NOUN'
>>> f()
'NOUN'
>>> def g():
...     return 'NOUN'
>>> g()
'NOUN'

Let’s see how default dictionaries could be used in a more substantial
language processing task.
Many language processing tasks — including tagging — struggle to correctly process
the hapaxes of a text. They can perform better with a fixed vocabulary and a
guarantee that no new words will appear. We can preprocess a text to replace
low-frequency words with a special «out of vocabulary» token UNK, with
the help of a default dictionary. (Can you work out how to do this without
reading on?)

We need to create a default dictionary that maps each word to its replacement.
The most frequent n words will be mapped to themselves.
Everything else will be mapped to UNK.

>>> alice = nltk.corpus.gutenberg.words('carroll-alice.txt')
>>> vocab = nltk.FreqDist(alice)
>>> v1000 = [word for (word, _) in vocab.most_common(1000)]
>>> mapping = defaultdict(lambda: 'UNK')
>>> for v in v1000:
...     mapping[v] = v
...
>>> alice2 = [mapping[v] for v in alice]
>>> alice2[:100]
['UNK', 'Alice', "'", 's', 'UNK', 'in', 'UNK', 'by', 'UNK', 'UNK', 'UNK',
'UNK', 'CHAPTER', 'I', '.', 'UNK', 'the', 'Rabbit', '-', 'UNK', 'Alice',
'was', 'beginning', 'to', 'get', 'very', 'tired', 'of', 'sitting', 'by',
'her', 'sister', 'on', 'the', 'UNK', ',', 'and', 'of', 'having', 'nothing',
'to', 'do', ':', 'once', 'or', 'twice', 'she', 'had', 'UNK', 'into', 'the',
'book', 'her', 'sister', 'was', 'UNK', ',', 'but', 'it', 'had', 'no',
'pictures', 'or', 'UNK', 'in', 'it', ',', "'", 'and', 'what', 'is', 'the',
'use', 'of', 'a', 'book', ",'", 'thought', 'Alice', "'", 'without',
'pictures', 'or', 'conversation', "?'" ...]
>>> len(set(alice2))
1001

3.5   Incrementally Updating a Dictionary

We can employ dictionaries to count occurrences, emulating the method
for tallying words shown in fig-tally.
We begin by initializing an empty defaultdict, then process each
part-of-speech tag in the text. If the tag hasn’t been seen before,
it will have a zero count by default. Each time we encounter a tag,
we increment its count using the += operator.

>>> from collections import defaultdict
>>> counts = defaultdict(int)
>>> from nltk.corpus import brown
>>> for (word, tag) in brown.tagged_words(categories='news', tagset='universal'):
...     counts[tag] += 1
...
>>> counts['NOUN']
30640
>>> sorted(counts)
['ADJ', 'PRT', 'ADV', 'X', 'CONJ', 'PRON', 'VERB', '.', 'NUM', 'NOUN', 'ADP', 'DET']

>>> from operator import itemgetter
>>> sorted(counts.items(), key=itemgetter(1), reverse=True)
[('NOUN', 30640), ('VERB', 14399), ('ADP', 12355), ('.', 11928), ...]
>>> [t for t, c in sorted(counts.items(), key=itemgetter(1), reverse=True)]
['NOUN', 'VERB', 'ADP', '.', 'DET', 'ADJ', 'ADV', 'CONJ', 'PRON', 'PRT', 'NUM', 'X']

The listing in 3.3 illustrates an important idiom for
sorting a dictionary by its values, to show words in decreasing
order of frequency. The first parameter of sorted() is the items
to sort, a list of tuples consisting of a POS tag and a frequency.
The second parameter specifies the sort key using a function itemgetter().
In general, itemgetter(n) returns a function that can be called on
some other sequence object to obtain the nth element, e.g.:

>>> pair = ('NP', 8336)
>>> pair[1]
8336
>>> itemgetter(1)(pair)
8336

The last parameter of sorted() specifies that the items should be returned
in reverse order, i.e. decreasing values of frequency.

There’s a second useful programming idiom at the beginning of
3.3, where we initialize a defaultdict and then use a
for loop to update its values. Here’s a schematic version:

Here’s another instance of this pattern, where we index words according to their last two letters:

>>> last_letters = defaultdict(list)
>>> words = nltk.corpus.words.words('en')
>>> for word in words:
...     key = word[-2:]
...     last_letters[key].append(word)
...
>>> last_letters['ly']
['abactinally', 'abandonedly', 'abasedly', 'abashedly', 'abashlessly', 'abbreviately',
'abdominally', 'abhorrently', 'abidingly', 'abiogenetically', 'abiologically', ...]
>>> last_letters['zy']
['blazy', 'bleezy', 'blowzy', 'boozy', 'breezy', 'bronzy', 'buzzy', 'Chazy', ...]

The following example uses the same pattern to create an anagram dictionary.
(You might experiment with the third line to get an idea of why this program works.)

>>> anagrams = defaultdict(list)
>>> for word in words:
...     key = ''.join(sorted(word))
...     anagrams[key].append(word)
...
>>> anagrams['aeilnrt']
['entrail', 'latrine', 'ratline', 'reliant', 'retinal', 'trenail']

Since accumulating words like this is such a common task,
NLTK provides a more convenient way of creating a defaultdict(list),
in the form of nltk.Index().

>>> anagrams = nltk.Index((''.join(sorted(w)), w) for w in words)
>>> anagrams['aeilnrt']
['entrail', 'latrine', 'ratline', 'reliant', 'retinal', 'trenail']

3.6   Complex Keys and Values

We can use default dictionaries with complex keys and values.
Let’s study the range of possible tags for a word, given the
word itself, and the tag of the previous word. We will see
how this information can be used by a POS tagger.

This example uses a dictionary whose default value for an entry
is a dictionary (whose default value is int(), i.e. zero).
Notice how we iterated over the bigrams of the tagged
corpus, processing a pair of word-tag pairs for each iteration .
Each time through the loop we updated our pos dictionary’s
entry for (t1, w2), a tag and its following word .
When we look up an item in pos we must specify a compound key ,
and we get back a dictionary object.
A POS tagger could use such information to decide that the
word right, when preceded by a determiner, should be tagged as ADJ.

3.7   Inverting a Dictionary

Dictionaries support efficient lookup, so long as you want to get the value for
any key. If d is a dictionary and k is a key, we type d[k] and
immediately obtain the value. Finding a key given a value is slower and more
cumbersome:

>>> counts = defaultdict(int)
>>> for word in nltk.corpus.gutenberg.words('milton-paradise.txt'):
...     counts[word] += 1
...
>>> [key for (key, value) in counts.items() if value == 32]
['brought', 'Him', 'virtue', 'Against', 'There', 'thine', 'King', 'mortal',
'every', 'been']

If we expect to do this kind of «reverse lookup» often, it helps to construct
a dictionary that maps values to keys. In the case that no two keys have
the same value, this is an easy thing to do. We just get all the key-value
pairs in the dictionary, and create a new dictionary of value-key
pairs. The next example also illustrates another way of initializing a
dictionary pos with key-value pairs.

>>> pos = {'colorless': 'ADJ', 'ideas': 'N', 'sleep': 'V', 'furiously': 'ADV'}
>>> pos2 = dict((value, key) for (key, value) in pos.items())
>>> pos2['N']
'ideas'

Let’s first make our part-of-speech dictionary a bit more realistic
and add some more words to pos using the dictionary update() method, to
create the situation where multiple keys have the same value. Then the
technique just shown for reverse lookup will no longer work (why
not?). Instead, we have to use append() to accumulate the words
for each part-of-speech, as follows:

>>> pos.update({'cats': 'N', 'scratch': 'V', 'peacefully': 'ADV', 'old': 'ADJ'})
>>> pos2 = defaultdict(list)
>>> for key, value in pos.items():
...     pos2[value].append(key)
...
>>> pos2['ADV']
['peacefully', 'furiously']

Now we have inverted the pos dictionary, and can look up any part-of-speech and find
all words having that part-of-speech. We can do the same thing even
more simply using NLTK’s support for indexing as follows:

>>> pos2 = nltk.Index((value, key) for (key, value) in pos.items())
>>> pos2['ADV']
['peacefully', 'furiously']

A summary of Python’s dictionary methods is given in 3.2.

Table 3.2:

Python’s Dictionary Methods: A summary of commonly-used methods and idioms
involving dictionaries.

4.1   The Default Tagger

The simplest possible tagger assigns the same tag to each token. This
may seem to be a rather banal step, but it establishes an important
baseline for tagger performance. In order to get the best result, we
tag each word with the most likely tag. Let’s find out which tag is
most likely (now using the unsimplified tagset):

>>> tags = [tag for (word, tag) in brown.tagged_words(categories='news')]
>>> nltk.FreqDist(tags).max()
'NN'

Now we can create a tagger that tags everything as NN.

>>> raw = 'I do not like green eggs and ham, I do not like them Sam I am!'
>>> tokens = nltk.word_tokenize(raw)
>>> default_tagger = nltk.DefaultTagger('NN')
>>> default_tagger.tag(tokens)
[('I', 'NN'), ('do', 'NN'), ('not', 'NN'), ('like', 'NN'), ('green', 'NN'),
('eggs', 'NN'), ('and', 'NN'), ('ham', 'NN'), (',', 'NN'), ('I', 'NN'),
('do', 'NN'), ('not', 'NN'), ('like', 'NN'), ('them', 'NN'), ('Sam', 'NN'),
('I', 'NN'), ('am', 'NN'), ('!', 'NN')]

Unsurprisingly, this method performs rather poorly.
On a typical corpus, it will tag only about an eighth of the tokens correctly,
as we see below:

>>> default_tagger.evaluate(brown_tagged_sents)
0.13089484257215028

Default taggers assign their tag to every single word, even words that
have never been encountered before. As it happens, once we have processed
several thousand words of English text, most new words will be nouns.
As we will see, this means that default taggers can help to improve the
robustness of a language processing system. We will return to them
shortly.

4.2   The Regular Expression Tagger

The regular expression tagger assigns tags to tokens on the basis of
matching patterns. For instance, we might guess that any word ending
in ed is the past participle of a verb, and any word ending with
‘s is a possessive noun. We can express these as a list of
regular expressions:

>>> patterns = [
...     (r'.*ing$', 'VBG'),                
...     (r'.*ed$', 'VBD'),                 
...     (r'.*es$', 'VBZ'),                 
...     (r'.*ould$', 'MD'),                
...     (r'.*'s$', 'NN$'),                
...     (r'.*s$', 'NNS'),                  
...     (r'^-?[0-9]+(.[0-9]+)?$', 'CD'),  
...     (r'.*', 'NN')                      
... ]

Note that these are processed in order, and the first one that matches is applied.
Now we can set up a tagger and use it to tag a sentence. Now its right about a fifth
of the time.

>>> regexp_tagger = nltk.RegexpTagger(patterns)
>>> regexp_tagger.tag(brown_sents[3])
[('``', 'NN'), ('Only', 'NN'), ('a', 'NN'), ('relative', 'NN'), ('handful', 'NN'),
('of', 'NN'), ('such', 'NN'), ('reports', 'NNS'), ('was', 'NNS'), ('received', 'VBD'),
("''", 'NN'), (',', 'NN'), ('the', 'NN'), ('jury', 'NN'), ('said', 'NN'), (',', 'NN'),
('``', 'NN'), ('considering', 'VBG'), ('the', 'NN'), ('widespread', 'NN'), ...]
>>> regexp_tagger.evaluate(brown_tagged_sents)
0.20326391789486245

The final regular expression «.*» is a catch-all that tags everything as a noun.
This is equivalent to the default tagger (only much less efficient).
Instead of re-specifying this as part of the regular expression tagger,
is there a way to combine this tagger with the default tagger? We
will see how to do this shortly.

4.3   The Lookup Tagger

A lot of high-frequency words do not have the NN tag.
Let’s find the hundred most frequent words and store their most likely tag.
We can then use this information as the model for a «lookup tagger»
(an NLTK UnigramTagger):

>>> fd = nltk.FreqDist(brown.words(categories='news'))
>>> cfd = nltk.ConditionalFreqDist(brown.tagged_words(categories='news'))
>>> most_freq_words = fd.most_common(100)
>>> likely_tags = dict((word, cfd[word].max()) for (word, _) in most_freq_words)
>>> baseline_tagger = nltk.UnigramTagger(model=likely_tags)
>>> baseline_tagger.evaluate(brown_tagged_sents)
0.45578495136941344

It should come as no surprise by now that simply
knowing the tags for the 100 most frequent words enables us to tag a large fraction of
tokens correctly (nearly half in fact).
Let’s see what it does on some untagged input text:

>>> sent = brown.sents(categories='news')[3]
>>> baseline_tagger.tag(sent)
[('``', '``'), ('Only', None), ('a', 'AT'), ('relative', None),
('handful', None), ('of', 'IN'), ('such', None), ('reports', None),
('was', 'BEDZ'), ('received', None), ("''", "''"), (',', ','),
('the', 'AT'), ('jury', None), ('said', 'VBD'), (',', ','),
('``', '``'), ('considering', None), ('the', 'AT'), ('widespread', None),
('interest', None), ('in', 'IN'), ('the', 'AT'), ('election', None),
(',', ','), ('the', 'AT'), ('number', None), ('of', 'IN'),
('voters', None), ('and', 'CC'), ('the', 'AT'), ('size', None),
('of', 'IN'), ('this', 'DT'), ('city', None), ("''", "''"), ('.', '.')]

Many words have been assigned a tag of None,
because they were not among the 100 most frequent words.
In these cases we would like to assign the default tag of NN.
In other words, we want to use the lookup table first,
and if it is unable to assign a tag, then use the default tagger,
a process known as backoff (5).
We do this by specifying one tagger as a parameter to the other,
as shown below. Now the lookup tagger will only store word-tag pairs
for words other than nouns, and whenever it cannot assign a tag to a
word it will invoke the default tagger.

>>> baseline_tagger = nltk.UnigramTagger(model=likely_tags,
...                                      backoff=nltk.DefaultTagger('NN'))

Let’s put all this together and write a program to create and
evaluate lookup taggers having a range of sizes, in 4.1.

def performance(cfd, wordlist):
    lt = dict((word, cfd[word].max()) for word in wordlist)
    baseline_tagger = nltk.UnigramTagger(model=lt, backoff=nltk.DefaultTagger('NN'))
    return baseline_tagger.evaluate(brown.tagged_sents(categories='news'))

def display():
    import pylab
    word_freqs = nltk.FreqDist(brown.words(categories='news')).most_common()
    words_by_freq = [w for (w, _) in word_freqs]
    cfd = nltk.ConditionalFreqDist(brown.tagged_words(categories='news'))
    sizes = 2 ** pylab.arange(15)
    perfs = [performance(cfd, words_by_freq[:size]) for size in sizes]
    pylab.plot(sizes, perfs, '-bo')
    pylab.title('Lookup Tagger Performance with Varying Model Size')
    pylab.xlabel('Model Size')
    pylab.ylabel('Performance')
    pylab.show()

Observe that performance initially increases rapidly as the model size grows, eventually
reaching a plateau, when large increases in model size yield little improvement
in performance. (This example used the pylab plotting package, discussed
in 4.8.)

5.1   Unigram Tagging

Unigram taggers are based on a simple statistical algorithm:
for each token, assign the tag that is most likely for
that particular token. For example, it will assign the tag JJ to any
occurrence of the word frequent, since frequent is used as an
adjective (e.g. a frequent word) more often than it is used as a
verb (e.g. I frequent this cafe).
A unigram tagger behaves just like a lookup tagger (4),
except there is a more convenient technique for setting it up,
called training. In the following code sample,
we train a unigram tagger, use it to tag a sentence, then evaluate:

>>> from nltk.corpus import brown
>>> brown_tagged_sents = brown.tagged_sents(categories='news')
>>> brown_sents = brown.sents(categories='news')
>>> unigram_tagger = nltk.UnigramTagger(brown_tagged_sents)
>>> unigram_tagger.tag(brown_sents[2007])
[('Various', 'JJ'), ('of', 'IN'), ('the', 'AT'), ('apartments', 'NNS'),
('are', 'BER'), ('of', 'IN'), ('the', 'AT'), ('terrace', 'NN'), ('type', 'NN'),
(',', ','), ('being', 'BEG'), ('on', 'IN'), ('the', 'AT'), ('ground', 'NN'),
('floor', 'NN'), ('so', 'QL'), ('that', 'CS'), ('entrance', 'NN'), ('is', 'BEZ'),
('direct', 'JJ'), ('.', '.')]
>>> unigram_tagger.evaluate(brown_tagged_sents)
0.9349006503968017

We train a UnigramTagger by specifying tagged sentence data as
a parameter when we initialize the tagger. The training process involves
inspecting the tag of each word and storing the most likely tag for any word
in a dictionary, stored inside the tagger.

5.2   Separating the Training and Testing Data

Now that we are training a tagger on some data, we must be careful not to test it on the
same data, as we did in the above example. A tagger that simply memorized its training data
and made no attempt to construct a general model would get a perfect score, but would also
be useless for tagging new text. Instead, we should split the data, training on 90% and
testing on the remaining 10%:

>>> size = int(len(brown_tagged_sents) * 0.9)
>>> size
4160
>>> train_sents = brown_tagged_sents[:size]
>>> test_sents = brown_tagged_sents[size:]
>>> unigram_tagger = nltk.UnigramTagger(train_sents)
>>> unigram_tagger.evaluate(test_sents)
0.811721...

Although the score is worse, we now have a better picture of the usefulness of
this tagger, i.e. its performance on previously unseen text.

5.3   General N-Gram Tagging

When we perform a language processing task based on unigrams, we are using
one item of context. In the case of tagging, we only consider the current
token, in isolation from any larger context. Given such a model, the best
we can do is tag each word with its a priori most likely tag.
This means we would tag a word such as wind with the same tag,
regardless of whether it appears in the context the wind or
to wind.

An n-gram tagger is a generalization of a unigram tagger whose context is
the current word together with the part-of-speech tags of the
n-1 preceding tokens, as shown in 5.1. The tag to be
chosen, t_n, is circled, and the context is shaded
in grey. In the example of an n-gram tagger shown in 5.1,
we have n=3; that is, we consider the tags of the two preceding words in addition
to the current word. An n-gram tagger
picks the tag that is most likely in the given context.

The NgramTagger class uses a tagged training corpus to determine which
part-of-speech tag is most likely for each context. Here we see
a special case of an n-gram tagger, namely a bigram tagger.
First we train it, then use it to tag untagged sentences:

>>> bigram_tagger = nltk.BigramTagger(train_sents)
>>> bigram_tagger.tag(brown_sents[2007])
[('Various', 'JJ'), ('of', 'IN'), ('the', 'AT'), ('apartments', 'NNS'),
('are', 'BER'), ('of', 'IN'), ('the', 'AT'), ('terrace', 'NN'),
('type', 'NN'), (',', ','), ('being', 'BEG'), ('on', 'IN'), ('the', 'AT'),
('ground', 'NN'), ('floor', 'NN'), ('so', 'CS'), ('that', 'CS'),
('entrance', 'NN'), ('is', 'BEZ'), ('direct', 'JJ'), ('.', '.')]
>>> unseen_sent = brown_sents[4203]
>>> bigram_tagger.tag(unseen_sent)
[('The', 'AT'), ('population', 'NN'), ('of', 'IN'), ('the', 'AT'), ('Congo', 'NP'),
('is', 'BEZ'), ('13.5', None), ('million', None), (',', None), ('divided', None),
('into', None), ('at', None), ('least', None), ('seven', None), ('major', None),
('``', None), ('culture', None), ('clusters', None), ("''", None), ('and', None),
('innumerable', None), ('tribes', None), ('speaking', None), ('400', None),
('separate', None), ('dialects', None), ('.', None)]

Notice that the bigram tagger manages to tag every word in a sentence it saw during
training, but does badly on an unseen sentence. As soon as it encounters a new word
(i.e., 13.5), it is unable to assign a tag. It cannot tag the following word
(i.e., million) even if it was seen during training, simply because it never
saw it during training with a None tag on the previous word. Consequently, the
tagger fails to tag the rest of the sentence. Its overall accuracy score is very low:

>>> bigram_tagger.evaluate(test_sents)
0.102063...

As n gets larger, the specificity of the contexts increases,
as does the chance that the data we wish to tag contains contexts that
were not present in the training data. This is known as the sparse
data problem, and is quite pervasive in NLP. As a consequence, there is a
trade-off between the accuracy and the coverage of our results (and
this is related to the precision/recall trade-off in information
retrieval).

5.4   Combining Taggers

One way to address the trade-off between accuracy and coverage is to
use the more accurate algorithms when we can, but to fall back on
algorithms with wider coverage when necessary. For example, we could
combine the results of a bigram tagger, a unigram tagger, and
a default tagger, as follows:

1. Try tagging the token with the bigram tagger.
2. If the bigram tagger is unable to find a tag for the token, try
   the unigram tagger.
3. If the unigram tagger is also unable to find a tag, use a default tagger.

Most NLTK taggers permit a backoff-tagger to be specified.
The backoff-tagger may itself have a backoff tagger:

>>> t0 = nltk.DefaultTagger('NN')
>>> t1 = nltk.UnigramTagger(train_sents, backoff=t0)
>>> t2 = nltk.BigramTagger(train_sents, backoff=t1)
>>> t2.evaluate(test_sents)
0.844513...

Note that we specify the backoff tagger when the tagger is
initialized so that training can take advantage of the backoff tagger.
Thus, if the bigram tagger would assign the same tag
as its unigram backoff tagger in a certain context,
the bigram tagger discards the training instance.
This keeps the bigram tagger model as small as possible. We can
further specify that a tagger needs to see more than one instance of a
context in order to retain it, e.g. nltk.BigramTagger(sents, cutoff=2, backoff=t1)
will discard contexts that have only been seen once or twice.

5.5   Tagging Unknown Words

Our approach to tagging unknown words still uses backoff to a regular-expression tagger
or a default tagger. These are unable to make use of context. Thus, if our tagger
encountered the word blog, not seen during training, it would assign it the same tag,
regardless of whether this word appeared in the context the blog or to blog.
How can we do better with these unknown words, or out-of-vocabulary items?

A useful method to tag unknown words based on context is to limit the vocabulary
of a tagger to the most frequent n words, and to replace every other word
with a special word UNK using the method shown in 3.
During training, a unigram tagger will probably learn that UNK is usually a noun.
However, the n-gram taggers will detect contexts in which it has some other tag.
For example, if the preceding word is to (tagged TO), then UNK
will probably be tagged as a verb.

5.6   Storing Taggers

Training a tagger on a large corpus may take a significant time. Instead of training a tagger
every time we need one, it is convenient to save a trained tagger in a file for later re-use.
Let’s save our tagger t2 to a file t2.pkl.

>>> from pickle import dump
>>> output = open('t2.pkl', 'wb')
>>> dump(t2, output, -1)
>>> output.close()

Now, in a separate Python process, we can load our saved tagger.

>>> from pickle import load
>>> input = open('t2.pkl', 'rb')
>>> tagger = load(input)
>>> input.close()

Now let’s check that it can be used for tagging.

>>> text = """The board's action shows what free enterprise
...     is up against in our complex maze of regulatory laws ."""
>>> tokens = text.split()
>>> tagger.tag(tokens)
[('The', 'AT'), ("board's", 'NN$'), ('action', 'NN'), ('shows', 'NNS'),
('what', 'WDT'), ('free', 'JJ'), ('enterprise', 'NN'), ('is', 'BEZ'),
('up', 'RP'), ('against', 'IN'), ('in', 'IN'), ('our', 'PP$'), ('complex', 'JJ'),
('maze', 'NN'), ('of', 'IN'), ('regulatory', 'NN'), ('laws', 'NNS'), ('.', '.')]

5.7   Performance Limitations

What is the upper limit to the performance of an n-gram tagger?
Consider the case of a trigram tagger. How many cases of part-of-speech ambiguity does it
encounter? We can determine the answer to this question empirically:

>>> cfd = nltk.ConditionalFreqDist(
...            ((x[1], y[1], z[0]), z[1])
...            for sent in brown_tagged_sents
...            for x, y, z in nltk.trigrams(sent))
>>> ambiguous_contexts = [c for c in cfd.conditions() if len(cfd[c]) > 1]
>>> sum(cfd[c].N() for c in ambiguous_contexts) / cfd.N()
0.049297702068029296

Thus, one out of twenty trigrams is ambiguous [EXAMPLES]. Given the
current word and the previous two tags, in 5% of cases there is more than one tag
that could be legitimately assigned to the current word according to
the training data. Assuming we always pick the most likely tag in
such ambiguous contexts, we can derive a lower bound on
the performance of a trigram tagger.

Another way to investigate the performance of a tagger is to study
its mistakes. Some tags may be harder than others to assign, and
it might be possible to treat them specially by pre- or post-processing
the data. A convenient way to look at tagging errors is the
confusion matrix. It charts expected tags (the gold standard)
against actual tags generated by a tagger:

>>> test_tags = [tag for sent in brown.sents(categories='editorial')
...                  for (word, tag) in t2.tag(sent)]
>>> gold_tags = [tag for (word, tag) in brown.tagged_words(categories='editorial')]
>>> print(nltk.ConfusionMatrix(gold_tags, test_tags))           

Based on such analysis we may decide to modify the tagset. Perhaps
a distinction between tags that is difficult to make can be dropped,
since it is not important in the context of some larger processing task.

Another way to analyze the performance bound on a tagger comes from
the less than 100% agreement between human annotators. [MORE]

In general, observe that the tagging process collapses distinctions:
e.g. lexical identity is usually lost when all personal pronouns are
tagged PRP. At the same time, the tagging process introduces
new distinctions and removes ambiguities: e.g. deal tagged as VB or NN.
This characteristic of collapsing certain distinctions and introducing new
distinctions is an important feature of tagging which
facilitates classification and prediction.
When we introduce finer distinctions in a tagset, an n-gram tagger gets
more detailed information about the left-context when it is deciding
what tag to assign to a particular word.
However, the tagger simultaneously has to do more work to classify the
current token, simply because there are more tags to choose from.
Conversely, with fewer distinctions (as with the simplified tagset),
the tagger has less information about context, and it has a smaller
range of choices in classifying the current token.

We have seen that ambiguity in the training data leads to an upper limit
in tagger performance. Sometimes more context will resolve the
ambiguity. In other cases however, as noted by (Church, Young, & Bloothooft, 1996), the
ambiguity can only be resolved with reference to syntax, or to world
knowledge. Despite these imperfections, part-of-speech tagging has
played a central role in the rise of statistical approaches to natural
language processing. In the early 1990s, the surprising accuracy of
statistical taggers was a striking demonstration that it was possible
to solve one small part of the language understanding problem, namely
part-of-speech disambiguation, without reference to deeper sources of
linguistic knowledge. Can this idea be pushed further? In 7.,
we shall see that it can.

7.1   Morphological Clues

The internal structure of a word may give useful clues as to the
word’s category. For example, -ness is a suffix
that combines with an adjective to produce a noun, e.g.
happy → happiness, ill → illness. So
if we encounter a word that ends in -ness, this is very likely
to be a noun. Similarly, -ment is a suffix that combines
with some verbs to produce a noun, e.g.
govern → government and establish → establishment.

English verbs can also be morphologically complex. For instance, the
present participle of a verb ends in -ing, and expresses
the idea of ongoing, incomplete action (e.g. falling, eating).
The -ing suffix also appears on nouns derived from verbs, e.g. the
falling of the leaves (this is known as the gerund).

7.2   Syntactic Clues

Another source of information is the typical contexts in which a word can
occur. For example, assume that we have already determined the
category of nouns. Then we might say that a syntactic criterion for an
adjective in English is that it can occur immediately before a noun,
or immediately following the words be or very. According
to these tests, near should be categorized as an adjective:

7.3   Semantic Clues

Finally, the meaning of a word is a useful clue as to its lexical
category. For example, the best-known definition of a noun is
semantic: «the name of a person, place or thing». Within modern linguistics,
semantic criteria for word classes are treated with suspicion, mainly
because they are hard to formalize. Nevertheless, semantic criteria
underpin many of our intuitions about word classes, and enable us to
make a good guess about the categorization of words in languages that
we are unfamiliar with. For example, if all we know about the Dutch word
verjaardag is that it means the same as the English word
birthday, then we can guess that verjaardag is a noun in
Dutch. However, some care is needed: although we might translate zij
is vandaag jarig as it’s her birthday today, the word
jarig is in fact an adjective in Dutch, and has no exact
equivalent in English.

7.4   New Words

All languages acquire new lexical items. A list of words recently
added to the Oxford Dictionary of English includes cyberslacker,
fatoush, blamestorm, SARS, cantopop, bupkis, noughties, muggle, and
robata. Notice that all these new words are nouns, and this is
reflected in calling nouns an open class. By contrast, prepositions
are regarded as a closed class. That is, there is a limited set of
words belonging to the class (e.g., above, along, at, below, beside,
between, during, for, from, in, near, on, outside, over, past,
through, towards, under, up, with), and membership of the set only
changes very gradually over time.

About this document…

UPDATED FOR NLTK 3.0.
This is a chapter from Natural Language Processing with Python,
by Steven Bird, Ewan Klein and Edward Loper,
Copyright © 2019 the authors.
It is distributed with the Natural Language Toolkit [http://nltk.org/],
Version 3.0, under the terms of the
Creative Commons Attribution-Noncommercial-No Derivative Works 3.0 United States License
[http://creativecommons.org/licenses/by-nc-nd/3.0/us/].

This document was built on
Wed 4 Sep 2019 11:40:48 ACST