Most common word python

I’d read a select amount of the file at a time. Split it into characters, and then split on each empty space. This is better than splitting on each new line as the file may be one line.

To do the former in Python 3 is fairly simple:

def read_chunks(file, chunk_size):
    while True:
        chunk = file.read(chunk_size)
        if not chunk:
            break
        yield from chunk

This has $O(text{chunk_size})$ memory usage, which is $O(1)$ as it’s a constant. It also correctly ends the iterator, when the file ends.

After this, you want to split the words up. Since we’re using str.split without any arguments, we should write just that method of splitting. We can use a fairly simple algorithm:

from string import whitespace

def split_whitespace(it):
    chunk = []
    for char in it:
        if char not in whitespace:
            chunk.append(char)
        elif chunk:
            yield tuple(chunk)
            chunk = []
    if chunk:
        yield tuple(chunk)

This has $O(k)$ memory, where $k$ is the size of the largest word. What we’d expect of a splitting function.

Finally we’d change from tuples to strings, using ''.join, and then use the collections.Counter. And split the word reading, and finding the most common into two different functions.

And so for an $O(k)$ memory usage version of your code, I’d use:

import sys
from collections import Counter
from string import whitespace


def read_chunks(file, chunk_size):
    while True:
        chunk = file.read(chunk_size)
        if not chunk:
            break
        yield from chunk


def split_whitespace(it):
    chunk = []
    for char in it:
        if char not in whitespace:
            chunk.append(char)
        elif chunk:
            yield tuple(chunk)
            chunk = []
    if chunk:
        yield tuple(chunk)


def read_words(path, chunk_size=1024):
    with open(path) as f:
        chars = read_chunks(f, chunk_size)
        tuple_words = split_whitespace(chars)
        yield from map(''.join, tuple_words)


def most_common_words(words, top=10):
    return dict(Counter(words).most_common(top))


if __name__ == '__main__':
    words = read_words(sys.argv[1])
    top_five_words = most_common_words(words, 5)

The challenge

Write a function that, given a string of text (possibly with punctuation and line-breaks), returns an array of the top-3 most occurring words, in descending order of the number of occurrences.

Assumptions:

• A word is a string of letters (A to Z) optionally containing one or more apostrophes (‘) in ASCII. (No need to handle fancy punctuation.)
• Matches should be case-insensitive, and the words in the result should be lowercased.
• Ties may be broken arbitrarily.
• If a text contains fewer than three unique words, then either the top-2 or top-1 words should be returned, or an empty array if a text contains no words.

Examples:

top_3_words("In a village of La Mancha, the name of which I have no desire to call to
mind, there lived not long since one of those gentlemen that keep a lance
in the lance-rack, an old buckler, a lean hack, and a greyhound for
coursing. An olla of rather more beef than mutton, a salad on most
nights, scraps on Saturdays, lentils on Fridays, and a pigeon or so extra
on Sundays, made away with three-quarters of his income.")
# => ["a", "of", "on"]

top_3_words("e e e e DDD ddd DdD: ddd ddd aa aA Aa, bb cc cC e e e")
# => ["e", "ddd", "aa"]

top_3_words("  //wont won't won't")
# => ["won't", "wont"]

Bonus points:

1. Avoid creating an array whose memory footprint is roughly as big as the input text.
2. Avoid sorting the entire array of unique words.

Test cases

from random import choice, randint, sample, shuffle, choices
import re
from collections import Counter


def check(s, this=None):                                            # this: only for debugging purpose
    returned_result = top_3_words(s) if this is None else this
    fs = Counter(w for w in re.findall(r"[a-zA-Z']+", s.lower()) if w != "'" * len(w))
    exp,expected_frequencies = map(list,zip(*fs.most_common(3))) if fs else ([],[])
    
    msg = ''
    wrong_words = [w for w in returned_result if not fs[w]]
    actual_freq = [fs[w] for w in returned_result]
    
    if wrong_words:
        msg = 'Incorrect match: words not present in the string. Your output: {}. One possible valid answer: {}'.format(returned_result, exp)
    elif len(set(returned_result)) != len(returned_result):
        msg = 'The result should not contain copies of the same word. Your output: {}. One possible output: {}'.format(returned_result, exp)
    elif actual_freq!=expected_frequencies:
        msg = "Incorrect frequencies: {} should be {}. Your output: {}. One possible output: {}".format(actual_freq, expected_frequencies, returned_result, exp)
    
    Test.expect(not msg, msg)



@test.describe("Fixed tests")
def fixed_tests():

    TESTS = (
    "a a a  b  c c  d d d d  e e e e e",
    "e e e e DDD ddd DdD: ddd ddd aa aA Aa, bb cc cC e e e",
    "  //wont won't won't ",
    "  , e   .. ",
    "  ...  ",
    "  '  ",
    "  '''  ",
    """In a village of La Mancha, the name of which I have no desire to cao
    mind, there lived not long since one of those gentlemen that keep a lance
    in the lance-rack, an old buckler, a lean hack, and a greyhound for
    coursing. An olla of rather more beef than mutton, a salad on most
    nights, scraps on Saturdays, lentils on Fridays, and a pigeon or so extra
    on Sundays, made away with three-quarters of his income.""",
    "a a a  b  c c X",
    "a a c b b",
    )
    for s in TESTS: check(s)
    
@test.describe("Random tests")
def random_tests():
    
    def gen_word():
        return "".join(choice("abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ'") for _ in range(randint(3, 10)))
    
    def gen_string():
        words = []
        nums = choices(range(1, 31), k=20)
        for _ in range(randint(0, 20)):
            words += [gen_word()] * nums.pop()
        shuffle(words)
        s = ""
        while words:
            s += words.pop() + "".join(choice("-,.?!_:;/ ") for _ in range(randint(1, 5)))
        return s
    
    @test.it("Tests")
    def it_1():
        for _ in range(100): check(gen_string())
            

The solution using Python

Option 1:

# use the Counter module
from collections import Counter
# use the regex module
import re

def top_3_words(text):
    # count the input, pass through a regex and lowercase it
    c = Counter(re.findall(r"[a-z']+", re.sub(r" '+ ", " ", text.lower())))
    # return the `most common` 3 items
    return [w for w,_ in c.most_common(3)]

Option 2:

def top_3_words(text):
    # loop through each character in the string
    for c in text:
        # if it's not alphanumeric or an apostrophe
        if not (c.isalpha() or c=="'"):
            # replace with a space
            text = text.replace(c,' ')
    # create some `list` variables
    words,counts,out = [],[],[]

    # loop through the words in the text
    for word in list(filter(None,text.lower().split())):
        # if in all, then continue
        if all([not c.isalpha() for c in word]):
            continue
        # if the word is in the words list
        if word in words:
            # increment the count
            counts[words.index(word)] += 1
        else:
            # otherwise create a new entry
            words.append(word); counts.append(0)

    # loop while bigger than 0 and less than 3
    while len(words)>0 and len(out)<3:
        # append the counts
        out.append(words.pop(counts.index(max(counts))).lower())
        counts.remove(max(counts))
    # return the counts
    return out

Option 3:

def top_3_words(text):
    wrds = {}
    for p in r'!"#$%&()*+,./:;<=>[email protected][]^_`{|}~-':
        text = text.replace(p, ' ')
    for w in text.lower().split():
        if w.replace("'", '') != '':
            wrds[w] = wrds.get(w, 0) + 1
    return [y[0] for y in sorted(wrds.items(), key=lambda x: x[1], reverse=True)[:3]]

Given Strings List, write a Python program to get word with most number of occurrences.

Example:

Input : test_list = [“gfg is best for geeks”, “geeks love gfg”, “gfg is best”] 
Output : gfg 
Explanation : gfg occurs 3 times, most in strings in total.

Input : test_list = [“geeks love gfg”, “geeks are best”] 
Output : geeks 
Explanation : geeks occurs 2 times, most in strings in total. 

Method #1 : Using loop + max() + split() + defaultdict()

In this, we perform task of getting each word using split(), and increase its frequency by memorizing it using defaultdict(). At last, max(), is used with parameter to get count of maximum frequency string.

The original list is : ['gfg is best for geeks', 'geeks love gfg', 'gfg is best']
Word with maximum frequency : gfg

Time Complexity: O(n*n)
Auxiliary Space: O(n)

Method #2 : Using list comprehension + mode()

In this, we get all the words using list comprehension and get maximum frequency using mode().

The original list is : ['gfg is best for geeks', 'geeks love gfg', 'gfg is best']
Word with maximum frequency : gfg

Method #3: Using list() and Counter()

• Append all words to empty list and calculate frequency of all words using Counter() function.
• Find max count and print that key.

Below is the implementation:

The original list is : ['gfg is best for geeks', 'geeks love gfg', 'gfg is best']
Word with maximum frequency : gfg

The time and space complexity for all the methods are the same:

Time Complexity: O(n²)

Space Complexity: O(n)

Method #4: Using Counter() and reduce()
Here is an approach to solve the problem using the most_common() function of the collections module’s Counter class and the reduce() function from the functools module:

The original list is:  ['gfg is best for geeks', 'geeks love gfg', 'gfg is best']
Word with most frequency:  gfg

Explanation:

We use the reduce() function to concatenate the list of all words from each string in the test_list.
We then create a Counter object from the list of all words to get a count of the frequency of each word.
Finally, we use the most_common() function to get the word with the highest frequency and return it.
Time complexity: O(n * k), where n is the number of strings in the test_list and k is the average number of words in each string.

Auxiliary Space: O(n * k), since we are storing the words in a list before creating a Counter object.

In this tutorial, you’ll learn how to use the Python Counter class from the collections module to count items. The Counter class provides an incredibly pythonic method to count items in lists, tuples, strings, and more. Because counting items is a common task in programming, being able to do this easily and elegantly is a useful skill for any Pythonista.

The Counter class provides a subclass to the Python dictionary, adding in many useful ways to easily count items in another object. For example, you can easily return the number of items, the most common item, and even undertake arithmetic on different Counter items.

By the end of this tutorial, you’ll have learned:

• How to use the Counter class to count items in Python
• How to get the most and least common items in a counter object
• How to add and subtract different Counter objects
• how to update Counter objects in Python

Understanding Python’s Collection Counter Class

The Python Counter class is an integral part of the collections module. The class provides incredibly intuitive and Pythonic methods to count items in an iterable, such as lists, tuples, or strings. This allows you to count the frequency of items within that iterable, including finding the most common item.

Let’s start by creating an empty Counter object. We first need to import the class from the collections module. Following that, we can instantiate the object:

# Creating an Empty Counter Object
from collections import Counter

counter = Counter()

Now that we have our first Counter object created, let’s explore some of the properties of the object. For example, we can check its type by using the type() function. We can also verify that the object is a subclass of the Python dictionary.

# Checking Attributes of the Python Counter Class
from collections import Counter

counter = Counter()

print('Type of counter is: ',type(counter))
print('Counter is a subclass of a dictionary: ', issubclass(Counter, dict))

# Returns:
# Type of counter is:  <class 'collections.Counter'>
# Counter is a subclass of a dictionary:  True

Now that you have an understanding of the Python Counter class, let’s get started with creating our first Counter object!

Creating a Counter Object in Python

Let’s create our first Python Counter object. We can pass in a string and the Counter object will return the counts of all the letters in that string.

The class takes only a single parameter, the item we want to count. Let’s see how we can use it:

# Creating Our First Counter
from collections import Counter

a_string = 'hello! welcome to datagy'
counter = Counter(a_string)

print(counter)

# Returns:
# Counter({'e': 3, 'l': 3, 'o': 3, ' ': 3, 't': 2, 'a': 2, 'h': 1, '!': 1, 'w': 1, 'c': 1, 'm': 1, 'd': 1, 'g': 1, 'y': 1})

By printing out our counter, we’re able to see that it returns a dictionary-like object. The items are sorted by their frequency of each item in the object. In this case, we can see that the letter 'e' exists three times in our string.

Accessing Counter Values in Python

Because the Counter object returns a subclass of a dictionary, we can use dictionary methods to access the counts of an item in that dictionary. Let’s see how we can access the number of times the letter 'a' appears in our string:

# Accessing Counts in a Counter Object
from collections import Counter

a_string = 'hello! welcome to datagy'
counter = Counter(a_string)

print(counter['a'])

# Returns: 2

We can see that the letter 'a' exists twice in our string. We can even access the counts of items that don’t exist in our object.

# Counting Items that Don't Exist
from collections import Counter

a_string = 'hello! welcome to datagy'
counter = Counter(a_string)

print(counter['z'])

# Returns: 0

In a normal Python dictionary, this would raise a KeyError. However, the Counter class has been designed to prevent this by overriding the default behavior.

Finding the Most Common Item in a Python Counter

The Counter class makes it easy to find the most common item in a given object. This can be done by applying the .most_common() method onto the object. Let’s see how we can find the most common item in our object:

# Finding the Most Common Item
from collections import Counter

a_string = 'hello! welcome to datagy'
counter = Counter(a_string)

print(counter.most_common())

# Returns: [('e', 3), ('l', 3), ('o', 3), (' ', 3), ('t', 2), ('a', 2), ('h', 1), ('!', 1), ('w', 1), ('c', 1), ('m', 1), ('d', 1), ('g', 1), ('y', 1)]

We can see that this returns a list of tuples that’s been ordered by placing the most common items first. Because of this, we can access the most common item by accessing the first index:

# Accessing the Most Common Item
from collections import Counter

a_string = 'hello! welcome to datagy'
counter = Counter(a_string)

print(counter.most_common()[0])

# Returns: ('e', 3)

Finding the Least Common Item in a Python Counter

Similarly, we can access the least common item by getting the last index:

# Accessing the Least Common Item
from collections import Counter

a_string = 'hello! welcome to datagy'
counter = Counter(a_string)

print(counter.most_common()[-1])

# Returns: ('y', 1)

Finding n Most Common Items in a Python Counter

Te method also allows you to pass in an integer that returns just that number of items. Say we wanted to get the three most common items, you could write:

# Getting n Number of Most Common Items
from collections import Counter

a_string = 'hello! welcome to datagy'
counter = Counter(a_string)

print(counter.most_common(3))

# Returns: [('e', 3), ('l', 3), ('o', 3)]

Updating Counter Values in Python

One of the great things about the Python Counter is that values can also be updated. This can be done using the .update() method. The method accepts another iterable, which will update the values in place.

Let’s see how we can first count items using the collection’s Counter class and then pass in another iterable to update our counts.

# Updating Counter Values in Python
from collections import Counter

a_list = [1,2,3,1,2,3,1,1,2,1]
counter = Counter(a_list)
print(counter)

counter.update([1,2,3,4])
print(counter)

# Returns:
# Counter({1: 5, 2: 3, 3: 2})
# Counter({1: 6, 2: 4, 3: 3, 4: 1})

We can see that the values were updated in place, with the values of the new item.

Deleting Counter Values in Python

It’s also very easy to delete an item from a Counter object. This can be useful when you either want to reset a value or simple find a way to remove an item from being counted.

You can delete an item from a Counter object by using the del keyword. Let’s load a Counter object and then delete a value:

# Deleting an Item from a Counter Object
from collections import Counter

a_list = [1,2,3,1,2,3,1,1,2,1]
counter = Counter(a_list)
print(counter)

del counter[1]
print(counter)

# Returns:
# Counter({1: 5, 2: 3, 3: 2})
# Counter({2: 3, 3: 2})

Arithmetic Operations on Counter Objects in Python

It’s equally easy to apply arithmetic operations like addition and subtraction on Counter objects. This allows you to combine Counter objects or find the difference between two items.

This can be done using the + and the - operators respectively. Let’s take a look at addition first:

# Adding 2 Counter Objects Together
from collections import Counter

counter1 = Counter([1,1,2,3,4])
counter2 = Counter([1,2,3,4])

print(counter1 + counter2)

# Returns:
# Counter({1: 3, 2: 2, 3: 2, 4: 2})

Now let’s subtract the two counters:

# Subtracting 2 Counter Objects
from collections import Counter

counter1 = Counter([1,1,2,3,4])
counter2 = Counter([1,2,3,4])

print(counter1 - counter2)

# Returns:
# Counter({1: 1})

Combining Counter Objects in Python

We can also combine Counter objects using the & and | operators. These serve very different purposes. Let’s break them down a bit:

• & will return the common positive minumum values
• | will return the positive maximum values

Let’s take a look at the & operator first:

# Finding Common Minimum Elements
from collections import Counter

counter1 = Counter([1,1,2,2,2,3,3,4])
counter2 = Counter([1,2,3,4,5])

print(counter1 & counter2)

# Returns:
# Counter({1: 1, 2: 1, 3: 1, 4: 1})

Now let’s take a look at the maximums between the two Counter objects:

# Finding Maximum Elements
from collections import Counter

counter1 = Counter([1,1,2,2,2,3,3,4])
counter2 = Counter([1,2,3,4,5])

print(counter1 | counter2)

# Returns:
# Counter({2: 3, 1: 2, 3: 2, 4: 1, 5: 1})

Finding the Most Common Word in a Python String

Before closing out the tutorial, let’s take a look at a practical example. We can use Python’s Counter class to count find the most common word in a string. Let’s load the Zen of Python and find the most common word in that string.

Before we pass the string into the Counter class, we need to split it. We can use the .split() method to split at any white-space character, including newlines. Then we can apply the .most_common() method and access the first item’s value by accessing the [0][0] item:

# Finding the Most Frequent Word in a String
from collections import Counter

text = """
Beautiful is better than ugly.
Explicit is better than implicit.
Simple is better than complex.
Complex is better than complicated.
Flat is better than nested.
Sparse is better than dense.
Readability counts.
Special cases aren't special enough to break the rules.
Although practicality beats purity.
Errors should never pass silently.
Unless explicitly silenced.
In the face of ambiguity, refuse the temptation to guess.
There should be one-- and preferably only one --obvious way to do it.
Although that way may not be obvious at first unless you're Dutch.
Now is better than never.
Although never is often better than *right* now.
If the implementation is hard to explain, it's a bad idea.
If the implementation is easy to explain, it may be a good idea.
Namespaces are one honking great idea -- let's do more of those!
"""

counter = Counter(text.split())
print(counter.most_common()[0][0])

# Returns: is

Conclusion

In this post, you learned how to use the Python collection’s Counter class. You started off by learning how the class can be used to create frequencies of an iterable object. You then learned how to find the counts of a particular item and how to find the most and least frequent item. You then learned how to update counts, as well as perform arithmetic on these count items.

Additional Resources

To learn more about related topics, check out the tutorials below:

• Python Defaultdict: Overview and Examples
• Python: Add Key:Value Pair to Dictionary
• Python Merge Dictionaries – Combine Dictionaries (7 Ways)
• Python: Sort a Dictionary by Values
• Official Documentation: Python collections Counter