Most used word in text

From Wikipedia, the free encyclopedia

Studies that estimate and rank the most common words in English examine texts written in English. Perhaps the most comprehensive such analysis is one that was conducted against the Oxford English Corpus (OEC), a massive text corpus that is written in the English language.

In total, the texts in the Oxford English Corpus contain more than 2 billion words.^[1] The OEC includes a wide variety of writing samples, such as literary works, novels, academic journals, newspapers, magazines, Hansard’s Parliamentary Debates, blogs, chat logs, and emails.^[2]

Another English corpus that has been used to study word frequency is the Brown Corpus, which was compiled by researchers at Brown University in the 1960s. The researchers published their analysis of the Brown Corpus in 1967. Their findings were similar, but not identical, to the findings of the OEC analysis.

According to The Reading Teacher’s Book of Lists, the first 25 words in the OEC make up about one-third of all printed material in English, and the first 100 words make up about half of all written English.^[3] According to a study cited by Robert McCrum in The Story of English, all of the first hundred of the most common words in English are of Old English origin,^[4] except for «people», ultimately from Latin «populus», and «because», in part from Latin «causa».

Some lists of common words distinguish between word forms, while others rank all forms of a word as a single lexeme (the form of the word as it would appear in a dictionary). For example, the lexeme be (as in to be) comprises all its conjugations (is, was, am, are, were, etc.), and contractions of those conjugations.^[5] These top 100 lemmas listed below account for 50% of all the words in the Oxford English Corpus.^[1]

100 most common words

A list of 100 words that occur most frequently in written English is given below, based on an analysis of the Oxford English Corpus (a collection of texts in the English language, comprising over 2 billion words).^[1] A part of speech is provided for most of the words, but part-of-speech categories vary between analyses, and not all possibilities are listed. For example, «I» may be a pronoun or a Roman numeral; «to» may be a preposition or an infinitive marker; «time» may be a noun or a verb. Also, a single spelling can represent more than one root word. For example, «singer» may be a form of either «sing» or «singe». Different corpora may treat such difference differently.

The number of distinct senses that are listed in Wiktionary is shown in the polysemy column. For example, «out» can refer to an escape, a removal from play in baseball, or any of 36 other concepts. On average, each word in the list has 15.38 senses. The sense count does not include the use of terms in phrasal verbs such as «put out» (as in «inconvenienced») and other multiword expressions such as the interjection «get out!», where the word «out» does not have an individual meaning.^[6] As an example, «out» occurs in at least 560 phrasal verbs^[7] and appears in nearly 1700 multiword expressions.^[8]

The table also includes frequencies from other corpora. Note that as well as usage differences, lemmatisation may differ from corpus to corpus – for example splitting the prepositional use of «to» from the use as a particle. Also the Corpus of Contemporary American English (COCA) list includes dispersion as well as frequency to calculate rank.

Parts of speech

The following is a very similar list, subdivided by part of speech.^[1] The list labeled «Others» includes pronouns, possessives, articles, modal verbs, adverbs, and conjunctions.

See also

• Basic English
• Frequency analysis, the study of the frequency of letters or groups of letters
• Letter frequencies
• Oxford English Corpus
• Swadesh list, a compilation of basic concepts for the purpose of historical-comparative linguistics
• Zipf’s law, a theory stating that the frequency of any word is inversely proportional to its rank in a frequency table

Word lists

• Dolch Word List, a list of frequently used English words
• General Service List
• Word lists by frequency

References

External links

With this online tool, you can find the most common words in any text. The program runs through all the words in the text and in the output, it prints the count of their occurrences. The information about the most popular words often gives clues about the topic, language, and purpose of the text. For example, if the most common words are «disco», «music», and «dance», then it’s most likely text about dancing. If the most common words are «the», «a», and «is», then the text is most likely written in the English language, and if the most common word are «di», «che», and «la», then the text is most likely in Italian. Even the information about a single word can tell a lot about the text. For example, if the most popular word in the text is «charity», then most likely the purpose of the text is to help those who need it. In addition to printing single-word statistics, this tool can also analyze the frequency of multi-word phrases in the text. You can choose to analyze combinations of two, three, or more words, and the tool will display the distribution of n-word groups. For example, if the input text is «Owls hoot in the dark.» then the program will generate four word pairs (also called word bigrams) – «Owls hoot», «hoot in», «in the», and «the dark», and if the word group size is 3 (called word trigrams), then there will be three word triplets – «Owls hoot in», «hoot in the», and «in the dark». You can also choose in the options (via the «Stop at Sentence Boundary») whether to create a stream of words from neighboring sentences to form joint groups or not. For example, if the input text is «Long cat is red. Short cat is black.», then with this option on, the bigrams would be «Long cat», «cat is», «is red», «Short cat», «cat is», «is black». But with this option off, the full-stop is ignored and the bigrams would be «Long cat», «cat is», «is red», «red Short», «Short cat», «cat is», «is black». In this example, the words maintained their sentence case but by enabling the «Ignore Word Case» option, you can analyze all words in lowercase. You can also exclude or replace punctuation marks in the text before analysis. For example, if a word is wrapped in parentheses «(owl)» then you can remove the parentheses by entering them in the «Punctuation to Delete» option. If a word contains internal punctuation, such as hyphenation in the word «full-scale», then you can replace the hyphen «-» with a space and analyze this word as two separate words «full» and «scale». In addition to the total number of words in the text, you can also display their usage percentage and print a fractional representation of each word’s number of uses relative to the total number of words in the text. Additionally, you can sort the output words alphabetically or by the usage counts. Textabulous!

The challenge

Write a function that, given a string of text (possibly with punctuation and line-breaks), returns an array of the top-3 most occurring words, in descending order of the number of occurrences.

Assumptions:

• A word is a string of letters (A to Z) optionally containing one or more apostrophes (‘) in ASCII. (No need to handle fancy punctuation.)
• Matches should be case-insensitive, and the words in the result should be lowercased.
• Ties may be broken arbitrarily.
• If a text contains fewer than three unique words, then either the top-2 or top-1 words should be returned, or an empty array if a text contains no words.

Examples:

top_3_words("In a village of La Mancha, the name of which I have no desire to call to
mind, there lived not long since one of those gentlemen that keep a lance
in the lance-rack, an old buckler, a lean hack, and a greyhound for
coursing. An olla of rather more beef than mutton, a salad on most
nights, scraps on Saturdays, lentils on Fridays, and a pigeon or so extra
on Sundays, made away with three-quarters of his income.")
# => ["a", "of", "on"]

top_3_words("e e e e DDD ddd DdD: ddd ddd aa aA Aa, bb cc cC e e e")
# => ["e", "ddd", "aa"]

top_3_words("  //wont won't won't")
# => ["won't", "wont"]

Bonus points:

1. Avoid creating an array whose memory footprint is roughly as big as the input text.
2. Avoid sorting the entire array of unique words.

Test cases

from random import choice, randint, sample, shuffle, choices
import re
from collections import Counter


def check(s, this=None):                                            # this: only for debugging purpose
    returned_result = top_3_words(s) if this is None else this
    fs = Counter(w for w in re.findall(r"[a-zA-Z']+", s.lower()) if w != "'" * len(w))
    exp,expected_frequencies = map(list,zip(*fs.most_common(3))) if fs else ([],[])
    
    msg = ''
    wrong_words = [w for w in returned_result if not fs[w]]
    actual_freq = [fs[w] for w in returned_result]
    
    if wrong_words:
        msg = 'Incorrect match: words not present in the string. Your output: {}. One possible valid answer: {}'.format(returned_result, exp)
    elif len(set(returned_result)) != len(returned_result):
        msg = 'The result should not contain copies of the same word. Your output: {}. One possible output: {}'.format(returned_result, exp)
    elif actual_freq!=expected_frequencies:
        msg = "Incorrect frequencies: {} should be {}. Your output: {}. One possible output: {}".format(actual_freq, expected_frequencies, returned_result, exp)
    
    Test.expect(not msg, msg)



@test.describe("Fixed tests")
def fixed_tests():

    TESTS = (
    "a a a  b  c c  d d d d  e e e e e",
    "e e e e DDD ddd DdD: ddd ddd aa aA Aa, bb cc cC e e e",
    "  //wont won't won't ",
    "  , e   .. ",
    "  ...  ",
    "  '  ",
    "  '''  ",
    """In a village of La Mancha, the name of which I have no desire to cao
    mind, there lived not long since one of those gentlemen that keep a lance
    in the lance-rack, an old buckler, a lean hack, and a greyhound for
    coursing. An olla of rather more beef than mutton, a salad on most
    nights, scraps on Saturdays, lentils on Fridays, and a pigeon or so extra
    on Sundays, made away with three-quarters of his income.""",
    "a a a  b  c c X",
    "a a c b b",
    )
    for s in TESTS: check(s)
    
@test.describe("Random tests")
def random_tests():
    
    def gen_word():
        return "".join(choice("abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ'") for _ in range(randint(3, 10)))
    
    def gen_string():
        words = []
        nums = choices(range(1, 31), k=20)
        for _ in range(randint(0, 20)):
            words += [gen_word()] * nums.pop()
        shuffle(words)
        s = ""
        while words:
            s += words.pop() + "".join(choice("-,.?!_:;/ ") for _ in range(randint(1, 5)))
        return s
    
    @test.it("Tests")
    def it_1():
        for _ in range(100): check(gen_string())
            

The solution using Python

Option 1:

# use the Counter module
from collections import Counter
# use the regex module
import re

def top_3_words(text):
    # count the input, pass through a regex and lowercase it
    c = Counter(re.findall(r"[a-z']+", re.sub(r" '+ ", " ", text.lower())))
    # return the `most common` 3 items
    return [w for w,_ in c.most_common(3)]

Option 2:

def top_3_words(text):
    # loop through each character in the string
    for c in text:
        # if it's not alphanumeric or an apostrophe
        if not (c.isalpha() or c=="'"):
            # replace with a space
            text = text.replace(c,' ')
    # create some `list` variables
    words,counts,out = [],[],[]

    # loop through the words in the text
    for word in list(filter(None,text.lower().split())):
        # if in all, then continue
        if all([not c.isalpha() for c in word]):
            continue
        # if the word is in the words list
        if word in words:
            # increment the count
            counts[words.index(word)] += 1
        else:
            # otherwise create a new entry
            words.append(word); counts.append(0)

    # loop while bigger than 0 and less than 3
    while len(words)>0 and len(out)<3:
        # append the counts
        out.append(words.pop(counts.index(max(counts))).lower())
        counts.remove(max(counts))
    # return the counts
    return out

Option 3:

def top_3_words(text):
    wrds = {}
    for p in r'!"#$%&()*+,./:;<=>[email protected][]^_`{|}~-':
        text = text.replace(p, ' ')
    for w in text.lower().split():
        if w.replace("'", '') != '':
            wrds[w] = wrds.get(w, 0) + 1
    return [y[0] for y in sorted(wrds.items(), key=lambda x: x[1], reverse=True)[:3]]

top_3_words("In a village of La Mancha, the name of which I have no desire to call to
mind, there lived not long since one of those gentlemen that keep a lance
in the lance-rack, an old buckler, a lean hack, and a greyhound for
coursing. An olla of rather more beef than mutton, a salad on most
nights, scraps on Saturdays, lentils on Fridays, and a pigeon or so extra
on Sundays, made away with three-quarters of his income.")
# => ["a", "of", "on"]

top_3_words("e e e e DDD ddd DdD: ddd ddd aa aA Aa, bb cc cC e e e")
# => ["e", "ddd", "aa"]

top_3_words("  //wont won't won't")
# => ["won't", "wont"]

WordCounter analyzes your text and tells you the most common words and phrases.

This tool helps you count words, bigrams, and trigrams in plain text. This is often the first step in quantitative text analysis.