Count word occurrences in python

Many times it is required to count the occurrence of each word in a text file. To achieve so, we make use of a dictionary object that stores the word as the key and its count as the corresponding value. We iterate through each word in the file and add it to the dictionary with a count of 1. If the word is already present in the dictionary we increment its count by 1. 

File sample.txt

First, we create a text file in which we want to count the words in Python. Let this file be sample.txt with the following contents

Mango banana apple pear
Banana grapes strawberry
Apple pear mango banana
Kiwi apple mango strawberry

Example 1: Count occurrences of each word in a given text file

Here, we use a Python loop to read each line, and from that line, we are converting each line to lower for the unique count and then split each word to count its number.

Output:

mango : 3
banana : 3
apple : 3
pear : 2
grapes : 1
strawberry : 2
kiwi : 1

Example 2: Count occurrences of specific words in a given text file

In this example, we will count the number of “apples” present in the text file.

Output:

Occurrences of the word apple: 2

Example 3: Count total occurrences of words in a given text file

In this example, we will count the total number of words present in a text file.

Output:

Total Number of Words: 15

Consider the files with punctuation

Sample.txt:

Mango! banana apple pear.
Banana, grapes strawberry.
Apple- pear mango banana.
Kiwi "apple" mango strawberry.

Code:

Output:

mango : 3
banana : 3
apple : 3
pear : 2
grapes : 1
strawberry : 2
kiwi : 1

Introduction

Counting the word frequency in a list element in Python is a relatively common task — especially when creating distribution data for histograms.

Say we have a list ['b', 'b', 'a'] — we have two occurrences of «b» and one of «a». This guide will show you three different ways to count the number of word occurrences in a Python list:

• Using Pandas and NumPy
• Using the count() Function
• Using the Collection Module’s Counter
• Using a Loop and a Counter Variable

In practice, you’ll use Pandas/Numpy, the count() function or a Counter as they’re pretty convenient to use.

Using Pandas and NumPy

The shortest and easiest way to get value counts in an easily-manipulable format (DataFrame) is via NumPy and Pandas. We can wrap the list into a NumPy array, and then call the value_counts() method of the pd instance (which is also available for all DataFrame instances):

import numpy as np
import pandas as pd

words = ['hello', 'goodbye', 'howdy', 'hello', 'hello', 'hi', 'bye']

pd.value_counts(np.array(words))

This results in a DataFrame that contains:

hello      3
goodbye    1
bye        1
howdy      1
hi         1
dtype: int64

You can access its values field to get the counts themselves, or index to get the words themselves:

df = pd.value_counts(np.array(words))

print('Index:', df.index)
print('Values:', df.values)

This results in:

Index: Index(['hello', 'goodbye', 'bye', 'howdy', 'hi'], dtype='object')

Values: [3 1 1 1 1]

Using the count() Function

The «standard» way (no external libraries) to get the count of word occurrences in a list is by using the list object’s count() function.

The count() method is a built-in function that takes an element as its only argument and returns the number of times that element appears in the list.

The complexity of the count() function is O(n), where n is the number of factors present in the list.

The code below uses count() to get the number of occurrences for a word in a list:

words = ['hello', 'goodbye', 'howdy', 'hello', 'hello', 'hi', 'bye']

print(f'"hello" appears {words.count("hello")} time(s)')
print(f'"howdy" appears {words.count("howdy")} time(s)')

This should give us the same output as before using loops:

"hello" appears 3 time(s)
"howdy" appears 1 time(s)

The count() method offers us an easy way to get the number of word occurrences in a list for each individual word.

Using the Collection Module’s Counter

The Counter class instance can be used to, well, count instances of other objects. By passing a list into its constructor, we instantiate a Counter which returns a dictionary of all the elements and their occurrences in a list.

From there, to get a single word’s occurrence you can just use the word as a key for the dictionary:

from collections import Counter

words = ['hello', 'goodbye', 'howdy', 'hello', 'hello', 'hi', 'bye']

word_counts = Counter(words)

print(f'"hello" appears {word_counts["hello"]} time(s)')
print(f'"howdy" appears {word_counts["howdy"]} time(s)')

This results in:

"hello" appears 3 time(s)
"howdy" appears 1 time(s)

Using a Loop and a Counter Variable

Ultimately, a brute force approach that loops through every word in the list, incrementing a counter by one when the word is found, and returning the total word count will work!

Of course, this method gets more inefficient as the list size grows, it’s just conceptually easy to understand and implement.

The code below uses this approach in the count_occurrence() method:

def count_occurrence(words, word_to_count):
    count = 0
    for word in words:
        if word == word_to_count:
          # update counter variable
            count = count + 1
    return count


words = ['hello', 'goodbye', 'howdy', 'hello', 'hello', 'hi', 'bye']
print(f'"hello" appears {count_occurrence(words, "hello")} time(s)')
print(f'"howdy" appears {count_occurrence(words, "howdy")} time(s)')

If you run this code you should see this output:

"hello" appears 3 time(s)
"howdy" appears 1 time(s)

Nice and easy!

Most Efficient Solution?

Naturally — you’ll be searching for the most efficient solution if you’re dealing with a large corpora of words. Let’s benchmark all of these to see how they perform.

The task can be broken down into finding occurrences for all words or a single word, and we’ll be doing benchmarks for both, starting with all words:

import numpy as np
import pandas as pd
import collections

def pdNumpy(words):
    def _pdNumpy():
        return pd.value_counts(np.array(words))
    return _pdNumpy

def countFunction(words):
    def _countFunction():
        counts = []
        for word in words:
            counts.append(words.count(word))
        return counts
    return _countFunction

def counterObject(words):
    def _counterObject():
        return collections.Counter(words)
    return _counterObject
    
import timeit

words = ['hello', 'goodbye', 'howdy', 'hello', 'hello', 'hi', 'bye']

print("Time to execute:n")
print("Pandas/Numpy: %ss" % timeit.Timer(pdNumpy(words)).timeit(1000))
print("count(): %ss" % timeit.Timer(countFunction(words)).timeit(1000))
print("Counter: %ss" % timeit.Timer(counterObject(words)).timeit(1000))

Which results in:

Time to execute:

Pandas/Numpy: 0.33886080000047514s
count(): 0.0009540999999444466s
Counter: 0.0019409999995332328s

The count() method is extremely fast compared to the other variants, however, it doesn’t give us the labels associated with the counts like the other two do.

If you need the labels — the Counter outperforms the inefficient process of wrapping the list in a NumPy array and then counting.

On the other hand, you can make use of DataFrame’s methods for sorting or other manipulation that you can’t do otherwise. Counter has some unique methods as well.

Ultimately, you can use the Counter to create a dictionary and turn the dictionary into a DataFrame as as well, to leverage the speed of Counter and the versatility of DataFrames:

df = pd.DataFrame.from_dict([Counter(words)]).T

If you don’t need the labels — count() is the way to go.

Alternatively, if you’re looking for a single word:

import numpy as np
import pandas as pd
import collections

def countFunction(words, word_to_search):
    def _countFunction():
        return words.count(word_to_search)
    return _countFunction

def counterObject(words, word_to_search):
    def _counterObject():
        return collections.Counter(words)[word_to_search]
    return _counterObject

def bruteForce(words, word_to_search):
    def _bruteForce():
        counts = []
        count = 0
        for word in words:
            if word == word_to_search:
              # update counter variable
                count = count + 1
            counts.append(count)
        return counts
    return _bruteForce
    
import timeit

words = ['hello', 'goodbye', 'howdy', 'hello', 'hello', 'hi', 'bye']

print("Time to execute:n")
print("count(): %ss" % timeit.Timer(countFunction(words, 'hello')).timeit(1000))
print("Counter: %ss" % timeit.Timer(counterObject(words, 'hello')).timeit(1000))
print("Brute Force: %ss" % timeit.Timer(bruteForce(words, 'hello')).timeit(1000))

Which results in:

Time to execute:

count(): 0.0001573999998072395s
Counter: 0.0019498999999996158s
Brute Force: 0.0005682000000888365s

The brute force search and count() methods outperform the Counter, mainly because the Counter inherently counts all words instead of one.

Conclusion

In this guide, we explored finding the occurrence of the word in a Python list, assessing the efficiency of each solution and weighing when each is more suitable.

In this tutorial, you’ll learn how to use Python to count the number of words and word frequencies in both a string and a text file. Being able to count words and word frequencies is a useful skill. For example, knowing how to do this can be important in text classification machine learning algorithms.

By the end of this tutorial, you’ll have learned:

• How to count the number of words in a string
• How to count the number of words in a text file
• How to calculate word frequencies using Python

Reading a Text File in Python

The processes to count words and calculate word frequencies shown below are the same for whether you’re considering a string or an entire text file. Because of this, this section will briefly describe how to read a text file in Python.

If you want a more in-depth guide on how to read a text file in Python, check out this tutorial here. Here is a quick piece of code that you can use to load the contents of a text file into a Python string:

# Reading a Text File in Python
file_path = '/Users/datagy/Desktop/sample_text.txt'

with open(file_path) as file:
    text = file.read()

I encourage you to check out the tutorial to learn why and how this approach works. However, if you’re in a hurry, just know that the process opens the file, reads its contents, and then closes the file again.

Count Number of Words In Python Using split()

One of the simplest ways to count the number of words in a Python string is by using the split() function. The split function looks like this:

# Understanding the split() function
str.split(
   sep=None     # The delimiter to split on
   maxsplit=-1  # The number of times to split
)

By default, Python will consider runs of consecutive whitespace to be a single separator. This means that if our string had multiple spaces, they’d only be considered a single delimiter. Let’s see what this method returns:

# Splitting a string with .split()
text = 'Welcome to datagy! Here you will learn Python and data science.'
print(text.split())

# Returns: ['Welcome', 'to', 'datagy!', 'Here', 'you', 'will', 'learn', 'Python', 'and', 'data', 'science.']

We can see that the method now returns a list of items. Because we can use the len() function to count the number of items in a list, we’re able to generate a word count. Let’s see what this looks like:

# Counting words with .split()
text = 'Welcome to datagy! Here you will learn Python and data science.'
print(len(text.split()))

# Returns: 11

Count Number of Words In Python Using Regex

Another simple way to count the number of words in a Python string is to use the regular expressions library, re. The library comes with a function, findall(), which lets you search for different patterns of strings.

Because we can use regular expression to search for patterns, we must first define our pattern. In this case, we want patterns of alphanumeric characters that are separated by whitespace.

For this, we can use the pattern w+, where w represents any alphanumeric character and the + denotes one or more occurrences. Once the pattern encounters whitespace, such as a space, it will stop the pattern there.

Let’s see how we can use this method to generate a word count using the regular expressions library, re:

# Counting words with regular expressions
import re
text = 'Welcome to datagy! Here you will learn Python and data science.'
print(len(re.findall(r'w+', text)))

# Returns: 11

Calculating Word Frequencies in Python

In order to calculate word frequencies, we can use either the defaultdict class or the Counter class. Word frequencies represent how often a given word appears in a piece of text.

Using defaultdict To Calculate Word Frequencies in Python

Let’s see how we can use defaultdict to calculate word frequencies in Python. The defaultdict extend on the regular Python dictionary by providing helpful functions to initialize missing keys.

Because of this, we can loop over a piece of text and count the occurrences of each word. Let’s see how we can use it to create word frequencies for a given string:

# Creating word frequencies with defaultdict
from collections import defaultdict
import re

text = 'welcome to datagy! datagy will teach data. data is fun. data data data!'

counts = defaultdict(int)
for word in re.findall('w+', text):
    counts[word] += 1

print(counts)

# Returns:
# defaultdict(<class 'int'>, {'welcome': 1, 'to': 1, 'datagy': 2, 'will': 1, 'teach': 1, 'data': 5, 'is': 1, 'fun': 1})

Let’s break down what we did here:

1. We imported both the defaultdict function and the re library
2. We loaded some text and instantiated a defaultdict using the int factory function
3. We then looped over each word in the word list and added one for each time it occurred

Using Counter to Create Word Frequencies in Python

Another way to do this is to use the Counter class. The benefit of this approach is that we can even easily identify the most frequent word. Let’s see how we can use this approach:

# Creating word frequencies with Counter
from collections import Counter
import re

text = 'welcome to datagy! datagy will teach data. data is fun. data data data!'
counts =  Counter(re.findall('w+', text))
print(counts)

# Returns:
# Counter({'data': 5, 'datagy': 2, 'welcome': 1, 'to': 1, 'will': 1, 'teach': 1, 'is': 1, 'fun': 1})

Let’s break down what we did here:

1. We imported our required libraries and classes
2. We passed the resulting list from the findall() function into the Counter class
3. We printed the result of this class

One of the perks of this is that we can easily find the most common word by using the .most_common() function. The function returns a sorted list of tuples, ordering the items from most common to least common. Because of this, we can simply access the 0th index to find the most common word:

# Finding the Most Common Word
from collections import Counter
import re

text = 'welcome to datagy! datagy will teach data. data is fun. data data data!'
counts =  Counter(re.findall('w+', text))
print(counts.most_common()[0])

# Returns:
# ('data', 5)

Conclusion

In this tutorial, you learned how to generate word counts and word frequencies using Python. You learned a number of different ways to count words including using the .split() method and the re library. Then, you learned different ways to generate word frequencies using defaultdict and Counter. Using the Counter method, you were able to find the most frequent word in a string.

Additional Resources

To learn more about related topics, check out the tutorials below:

• Python str.split() – Official Documentation
• Python Defaultdict: Overview and Examples
• Python: Count Number of Occurrences in List (6 Ways)
• Python: Count Number of Occurrences in a String (4 Ways!)