What is the word vector

Word vectors represent a significant leap forward in advancing our ability to analyze relationships across words, sentences, and documents. In doing so, they advance technology by providing machines with much more information about words than has previously been possible using traditional representations of words. It is word vectors that make technologies such as speech recognition and machine translation possible. There are many excellent explanations of word vectors, but in this one, I want to make the concept accessible to data and research people who aren’t very familiar with natural language processing (NLP).

What Are Word Vectors?

A word vector is an attempt to mathematically represent the meaning of a word. A computer goes through the text and calculates how often words show up next to each other. It is useful, first of all, to consider why word vectors are considered such a leap forward from traditional representations of words.

Traditional approaches to NLP, such as one-hot encoding and bag-of-words models (i.e. using dummy variables to represent the presence or absence of a word in an observation, i.e. a sentence), while useful for some machine learning (ML) tasks, do not capture information about a word’s meaning or context. This means that potential relationships, such as contextual closeness, are not captured across collections of words. For example, a one-hot encoding cannot capture simple relationships, such as determining that the words «dog» and «cat» both refer to animals that are often discussed in the context of household pets. Such encodings often provide sufficient baselines for simple NLP tasks (for example, email spam classifiers), but lack the sophistication for more complex tasks such as translation and speech recognition. In essence, traditional approaches to NLP such as one-hot encodings do not capture syntactic (structure) and semantic (meaning) relationships across collections of words and, therefore, represent language in a very naive way.

In contrast, word vectors represent words as multidimensional continuous floating point numbers where semantically similar words are mapped to proximate points in geometric space. In simpler terms, a word vector is a row of real-valued numbers (as opposed to dummy numbers) where each point captures a dimension of the word’s meaning and where semantically similar words have similar vectors. This means that words such as wheel and engine should have similar word vectors to the word car (because of the similarity of their meanings), whereas the word banana should be quite distant. Put differently, words that are used in a similar context will be mapped to a proximate vector space (we will get to how these word vectors are created below). The beauty of representing words as vectors is that they lend themselves to mathematical operators. For example, we can add and subtract vectors — the canonical example here is showing that by using word vectors we can determine that:

• king — man + woman = queen

In other words, we can subtract one meaning from the word vector for king (i.e. maleness), add another meaning (femaleness), and show that this new word vector (king — man + woman) maps most closely to the word vector for queen.

The numbers in the word vector represent the word’s distributed weight across dimensions. In a simplified sense, each dimension represents a meaning and the word’s numerical weight on that dimension captures the closeness of its association with and to that meaning. Thus, the semantics of the word are embedded across the dimensions of the vector.

A Simplified Representation of Word Vectors

In the figure, we are imagining that each dimension captures a clearly defined meaning. For example, if you imagine that the first dimension represents the meaning or concept of «animal,» then each word’s weight on that dimension represents how closely it relates to that concept.

This is quite a large simplification of word vectors as the dimensions do not hold such clearly defined meanings, but it is a useful and intuitive way to wrap your head around the concept of word vector dimensions.

We create a list of words, apply spaCy’s parser, extract the vector for each word, stack them together, and then extract two-principal components for visualization purposes.

import numpy as np
import spacy
from sklearn.decomposition import PCA
nlp = spacy.load("en")
animals = "dog cat hamster lion tiger elephant cheetah monkey gorilla antelope rabbit mouse rat zoo home pet fluffy wild domesticated"
animal_tokens = nlp(animals)
animal_vectors = np.vstack([word.vector for word in animal_tokens if word.has_vector])
pca = PCA(n_components=2)
animal_vecs_transformed = pca.fit_transform(animal_vectors)
animal_vecs_transformed = np.c_[animals.split(), animal_vecs_transformed]

Here, we simply extract vectors for different animals and words that might be used to describe some of them. As mentioned in the beginning, word vectors are amazingly powerful because they allow us (and machines) to identify similarities across different words by representing them in a continuous vector space. You can see here how the vectors for animals like «lion,» «tiger,» «cheetah,» and «elephant» are very close together. This is likely because they are often discussed in similar contexts; for example, these animals are big, wild and, potentially dangerous — indeed, the descriptive word «wild» maps quite closely to this group of animals.

Similar words are mapped together in the vector space. Notice how close «cat» and «dog» are to «pet,» how clustered «elephant,» «lion,» and «tiger» are, and how descriptive words also cluster together.

What is also interesting here is how closely the words «wild,» «zoo,» and «domesticated» map to one another. It makes sense given that they are words that are frequently used to describe animals, but highlights the amazing power of word vectors!

Where Do Word Vectors Come From?

An excellent question at this point is, Where do these dimensions and weights come from?! There are two common ways through which word vectors are generated:

1. Counts of word/context co-occurrences
2. Predictions of context given word (skip-gram neural network models, i.e. word2vec)

Note: Below, I describe a high-level word2vec approach to generating word vectors, but a good overview of the count/co-occurence approach can be found here.

Both approaches to generating word vectors build on Firth’s (1957) distributional hypothesis which states:

«You shall know a word by the company it keeps.»

Put differently, words that share similar contexts tend to have similar meanings. The context of a word in a practical sense refers to its surrounding word(s) and word vectors are (typically) generated by predicting the probability of a context given a word. Put differently, the weights that comprise a word vector are learned by making predictions on the probability that other words are contextually close to a given word. This is akin to attempting to fill in the blanks around some given input word. For example, given the input sequence, «The fluffy dog barked as it chased a cat,» the two-window (two-words preceding and proceeding the focal word) context for the words «dog» and «barked» would look like:

I don’t wish to delve into the mathematical details how neural networks learn word embeddings too much, as people much more qualified to do so have explicated this already. In particular, these posts have been helpful to me when trying to understand how word vectors are learned:

1. Deep Learning, NLP, and Representations
2. The Amazing Power of Word Vectors
3. Word2Vec Tutorial: The Skip-Gram Model

It is useful, however, to touch on the workings of the word2vec model given its popularity and usefulness. A word2vec model is simply a neural network with a single hidden layer that is designed to reconstruct the context of words by estimating the probability that a word is «close» to another word given as input.

The model is trained on word, context pairings for every word in the corpus, i.e.:

• (DOG, THE)(DOG), FLUFFY(DOG, BARKED)(DOG, AS)

Note that this is technically a supervised learning process, but you do not need labeled data — the labels (the targets/dependent variables) are generated from the words that form the context of a focal word. Thus, using the window function the model learns the context in which words are used. In this simple example, the model will learn that «fluffy» and «barked» are used in the context (as defined by the window length) of the word «dog.»

One of the fascinating things about word vectors created by word2vec models is that they are the side effects of a predictive task, not its output. In other words, a word vector is not predicted, (it is context probabilities that are predicted), the word vector is a learned representation of the input that is used on the predictive task — i.e. predicting a word given a context. The word vector is the model’s attempt to learn a good numerical representation of the word in order to minimize the loss (error) of its predictions. As the model iterates, it adjusts its neurons’ weights in an attempt to minimize the error of its predictions and in doing so, it gradually refines its representation of the word. In doing so, the word’s «meaning» becomes embedded in the weight learned by each neuron in the hidden layer of the network.

A word2vec model, therefore, accepts as input a single word (represented as a one-hot encoding amongst all words in the corpus) and the model attempts to predict the probability that a randomly chosen word in the corpus is at a nearby position to the input word. This means that for every input word there are n output probabilities, where n is equal to the total size of the corpus. The magic here is that the training process includes only the word’s context, not all words in the corpus. This means in our simple example above, given the word «dog» as input, «barked» will have a higher probability estimate than «cat» because it is closer in context, i.e. it is learned in the training process. Put differently, the model attempts to predict the probability that other words in the corpus belong to the context of the input word. Therefore, given the sentence above («The fluffy dog barked as it chased a cat») as input a run of the model would look like this:

Note: This conceptual NN is a close friend of the diagram in Chris McCormick’s blog post linked to above.

The value of going through this process is to extract the weights that have been learned by the neurons of the model’s hidden layer. It is these weights that form the word vector, i.e. if you have a 300-neuron hidden layer, you will create a 300-dimension word vector for each word in the corpus. The output of this process, therefore, is a word-vector mapping of size n-input words * n-hidden layer neurons.

Next Up

Word vectors are an amazingly powerful concept and a technology that will enable significant breakthroughs in NLP applications and research. They also highlight the beauty of neural network deep learning and, particularly, the power of learned representations of input data in hidden layers. In my next post, I will be using word vectors in a convolutional neural network for a classification task. This will highlight word vectors in practice, as well as how to bring pre-trained word vectors into a Keras model.

How do you define word vectors? In this post, I will introduce you to the concept of word vectors. We’ll go over different types of word embeddings, and more importantly, how word vectors function. We’ll then be able to see the impact of word vectors on SEO, which will lead us to understand how Schema.org markup for structured data can help you take advantage of word vectors in SEO.

Keep reading this post if you wish to learn more about these topics.

Let’s dive right in.

Word vectors (also called Word embeddings) are a type of word representation that allows words with similar meanings to have an equal representation.

In simple terms: A word vector is a vector representation of a particular word.

According to Wikipedia:
It is a technique used in natural language processing (NLP) for representing words for text analysis, typically as a real-valued vector that encodes the meaning of the word so that words that are close in vector space are likely to have similar meanings.

The following example will help us to understand this better:

Look at these similar sentences:

Have a good day. and Have a great day.

They barely have a different meaning. If we construct an exhaustive vocabulary (let’s call it V), it would have V = {Have, a, good, great, day} combining all words. We could encode the word as follows.
The vector representation of a word may be a one-hot encoded vector where 1 represents the position where the word exists and 0 represents the rest
Have = [1,0,0,0,0]
a=[0,1,0,0,0]
good=[0,0,1,0,0]
great=[0,0,0,1,0]
day=[0,0,0,0,1]

Suppose our vocabulary has only five words: King, Queen, Man, Woman, and Child. We could encode the words as:

King = [1,0,0,0,0]
Queen = [0,1,0,0,0]
Man = [0,0,1,00]
Woman = [0,0,0,1,0]
Child = [0,0,0,0,1]

Types of word Embedding (Word Vectors)

Word Embedding is one such technique in which vectors represent text. Here are some of the more popular types of word Embedding:

• Frequency-based Embedding
• Prediction based Embedding

We will not go deep into Frequency-based Embedding and Prediction-based Embedding here, but you may find the following guides helpful to understand both:

An Intuitive Understanding of Word Embeddings and Quick Introduction to Bag-of-Words (BOW) and TF-IDF for Creating Features from Text

A brief introduction to WORD2Vec

While Frequency-based Embedding has gained popularity, there is still a void in understanding the context of words and limited in their word representations.

Prediction-based Embedding (WORD2Vec) was created, patented, and introduced to the NLP community in 2013 by a team of researchers led by Tomas Mikolov at Google.

According to Wikipedia, the word2vec algorithm uses a neural network model to learn word associations from a large corpus of text (large and structured set of texts).

Once trained, such a model can detect synonymous words or suggest additional words for a partial sentence. For example, with Word2Vec, you can easily create such results: King – man + woman = Queen, which was considered an almost magical result.

Image source: Tensorflow

• [king] – [man] + [woman] ~= [queen] (another way of thinking about this is that [king] – [queen] is encoding just the gendered part of [monarch])
• [walking] – [swimming] + [swam] ~= [walked] (or [swam] – [swimming] is encoding just the “past-tense-ness” of the verb)
• [madrid] – [spain] + [france] ~= [paris] (or [madrid] – [spain] ~= [paris] – [france] which is presumably roughly “capital”)

Source: Brainslab Digital

I know this is a little technical, but Stitch Fix put together a fantastic post about semantic relationships and word vectors.

The Word2Vec algorithm is not a single algorithm but a combination of two techniques that uses a few AI methods to bridge human comprehension and machine comprehension. This technique is essential in‌ ‌solving‌ ‌many‌ ‌NLP‌ ‌problems.

These two techniques are:

• – CBOW (Continuous bag of words) or CBOW model
• – Skip-gram model.

Both‌ are shallow neural networks that provide probabilities for words and have been proven helpful in tasks such as word comparison and word analogy.

How word vectors and word2vecs works

Word Vector is an AI model developed by Google, and it helps us solve very complex NLP tasks.

“Word Vector models have one central goal that you should know:

It is an algorithm that helps Google in detecting semantic relationships between words.”

Each word is encoded in a vector (as a number represented in multiple dimensions) to match vectors of words that appear in a similar context. Hence a dense vector is formed for the text.

These vector models map semantically similar phrases to nearby points based on equivalence, similarities, or relatedness of ideas and language

Word2Vec- How does it work?

Image source: Seopressor

Pros and Cons of Word2Vec

We have seen that Word2vec is a very effective technique for generating distributional similarity. I have listed some of its other advantages here:

• There is no difficulty understanding Word2vec concepts. Word2Vec is not so complex that you aren’t aware of what is happening behind the scenes.
• Word2Vec’s architecture is very powerful and easy to use. Compared to other techniques, it is fast to train.
• Training is almost entirely automated here, so human-tagged data is no longer required.
• This technique works for both small and large datasets. As a result, it is an easy-to-scale model.
• If you know the concepts, you can easily replicate the entire concept and algorithm.
• It captures semantic similarity exceptionally well.
• Accurate and computationally efficient
• Since this approach is unsupervised, it is very time-saving in terms of effort.

Challenges of Word2Vec

The Word2vec concept is very efficient, but you may find a few points a bit challenging. Here are a few of the most common challenges.

• When developing a word2vec model for your dataset, debugging can be a major challenge, as the word2vec model is easy to develop but hard to debug.
• It does not deal with ambiguities. So, in the case of words with multiple meanings, Embedding will reflect the average of these meanings in vector space.
• Unable to handle unknown or OOV words: The biggest problem with word2vec is the inability to handle unknown or out-of-vocabulary (OOV) words.

Word Vectors: A Game-changer in Search Engine Optimization?

Many SEO experts believe Word Vector affects a website’s ranking in search engine results.

Over the past five years, Google has introduced two algorithm updates that put a clear focus on content quality and language comprehensiveness.

Let’s take a step back and talk about the updates:

Hummingbird

In 2013, Hummingbird gave search engines the capability of semantic analysis. By utilizing and incorporating semantics theory in their algorithms, they opened a new path to the world of search.

Google Hummingbird was the biggest change to the search engine since Caffeine in 2010. It gets its name from being “precise and fast”.

According to Search Engine Land, Hummingbird pays more attention to each word in a query, ensuring that the entire query is considered, rather than just particular words.

The main goal of Hummingbird was to deliver better results by understanding the context of the query rather than returning results for specific keywords.

“Google Hummingbird was released in September 2013.”

RankBrain

In 2015, Google announced RankBrain, a strategy that incorporated artificial intelligence (AI).

RankBrain is an algorithm that helps Google break down complex search queries into simpler ones. RankBrain converts search queries from “human” language into a language that Google can easily understand.

Google confirmed the use of RankBrain on 26 October 2015 in an article published by Bloomberg.

BERT

On 21 October 2019, BERT began rolling out in Google’s search system
BERT stands for Bidirectional Encoder Representations from Transformers, a neural network-based technique used by Google for pre-training in natural language processing (NLP).

In short, BERT helps computers understand language more like humans, and it is the biggest change in search since Google introduced RankBrain.

It is not a replacement for RankBrain, but rather an added method for understanding content and queries.

Google uses BERT in its ranking system as an addition. The RankBrain algorithm still exists for some queries and will continue to exist. But when Google feels that BERT can better understand a query, they will use that.

For more information on BERT, check out this post by Barry Schwartz, as well as Dawn Anderson’s in-depth dive.

Rank your site with Word Vectors

I’m assuming you already have created and published unique content, and even after polishing it over and over again, it does not improve your ranking or traffic.
Do you wonder why this is happening to you?

It might be because you didn’t include Word Vector: Google’s AI model.

• The first step is to Identify the Word Vectors of the top 10 SERP rankings for your niche.
• Know what keywords your competitors are using and what you might be overlooking.

By applying Word2Vec, which takes advantage of advanced Natural Language processing techniques and machine learning framework, you will be able to see everything in detail.

But these are possible if you know the machine learning and NLP techniques, but we can apply word vectors in the content using the following tool:

WordGraph, World’s First Word Vector Tool

This artificial intelligence tool is created with Neural Networks for Natural Language Processing and trained with Machine Learning.

Based on Artificial Intelligence, WordGraph analyzes your content and helps you improve its relevance to the Top 10 ranking websites.

It suggests keywords that are mathematically and contextually related to your main keyword.
Personally, I pair it up with BIQ, a powerful SEO tool that works well with WordGraph.

Add your content to the content intelligence tool built into Biq. It will show you a whole list of on-page SEO tips that you can add if you want to rank in the top position.

You can see how content intelligence works in this example. The lists will help you master on-page SEO and rank using actionable methods!

How to Supercharge Word Vectors: Using Structured Data Markup

Schema markup, or structured data, is a type of code (written in JSON, Java-Script Object Notation) created using schema.org vocabulary that helps search engines to crawl, organize, and display your content.

How to add structured data

Structured data can be easily added to your website by adding an inline script in your html
An example below shows how to define your organization’s structured data in the simplest format possible.

To generate the Schema Markup, I use this Schema Markup Generator (JSON-LD).

Here is the live example of schema markup for https://www.telecloudvoip.com/. Check the source code and search for JSON.

After the schema markup code is created, use Google’s Rich Results Test to see if the page supports rich results.
You can also use the Semrush Site Audit tool to explore Structured Data items for each URL and identify which pages are eligible for being in Rich Results.

Why is Structured Data Important for SEO?

Structured Data is important for SEO because it helps Google to understand what your website and pages are about, resulting in a more accurate ranking of your content.
Structured Data improves both the Search Bot’s experience as well as the user’s experience by improving the SERP (search engine result pages) with more information and accuracy.
To see the impact in Google search, go to Search Console and under Performance > Search Result > Search Appearance, you can view a breakdown of all the rich result types like “videos” and “FAQs” and see the organic impressions and clicks they drove for your content.

The following are some advantages of structured data:

• Structured data support semantic search
• It also supports your E‑A-T (expertise, authoritativeness, and trust)
• Having structured data can also increase conversion rates, since more people will see your listings, which increases the likelihood that they will buy from you.
• Using structured data the search engines are better able to understand your brand, your website, and your content.
• It will be easier for search engines to distinguish between contact pages, product descriptions, recipe pages, events pages, and customer reviews.
• With the help of structured data Google builds a better, more accurate knowledge graph and knowledge panel about your brand.
• These improvements can result in more organic impressions and organic clicks.

Structured data is currently used by Google to enhance search results. When people search for your web pages using keywords, structured data can help you get better results. Search engines will notice your content more if we add Schema markup.
You can implement schema markup on a number of different items. Listed below are a few areas where schema can be applied:

• Articles
• Blog Posts
• News Articles
• Events
• Products
• Videos
• Services
• Reviews
• Aggregate Ratings
• Restaurants
• Local businesses

Here’s a full list of the items you can mark up with schema.

Structured Data with Entity Embeddings

The term “entity” refers to a representation of any type of object, concept, or subject. An entity can be a person, movie, book, idea, place, company, or event.
While machines can’t really understand words, with entity embeddings, they are able to easily understand the relationship between king – queen = husband – wife
Entity embeddings perform better than one-hot encodings

The word vector algorithm is used by Google to discover semantic relationships between words, and when combined with structured data, we end up with a semantically enhanced web.

By using structured data, you are contributing to a more semantic web. This is an enhanced web where we describe the data in a machine-readable format.

Structured semantic data on your website helps search engines match your content with the right audience. The use of NLP, Machine Learning and Deep Learning helps reduce the gap between what people search for and what titles are available.

Final Thoughts

As you now understand the concept of word vectors and its importance, you can make your organic search strategy more effective and more efficient by utilizing word vectors, entity embeddings and structured semantic data.
In order to achieve the highest ranking, traffic, and conversions, you must use word vectors, entity embeddings and structured semantic data to demonstrate to Google that the content on your webpage is accurate, precise, and trustworthy.

In this post we will look at representing text documents with word vectors, which are vectors of numbers that represent the meaning of a word.

Then we will write a custom Scikit-learn transformer class for the word vector features — similar to TfidfVectorizer or CountVectorizer — which can be plugged into a pipeline.

What are word vectors?

Word vectors, or word embeddings, are vectors of numbers that provide information about the meaning of a word, as well as its context.

You can get the semantic similarity of two words by comparing their word vectors.

Even if you’re not familiar with word vectors, you may have heard of a couple of popular algorithms for obtaining vectors for various words.

• GloVe
• word2vec

There are pre-trained models that you can download to access word vectors, and if you are using Spacy, GloVe vectors are made available in the larger models.

Accessing word vectors in Spacy

With Spacy you can easily get vectors of words, as well as sentences.

I’m assuming at least some familiarity with Spacy in this post.

Note that a small Spacy model — ending in sm, such as en_core_web_sm, will not have built-in vectors, so you will need a larger model to use them.

python -m spacy download en_core_web_lg

Vectors are made available in Spacy Token, Doc and Span objects.

import spacy

nlp = spacy.load("en_core_web_lg")

With Spacy, you can get vectors for individual words, as well as sentences.

The vector will be a one-dimensional Numpy array of float numbers.

For example, take the word hat.

First you could check if the word has a vector.

hat = nlp("hat")
hat.has_vector 

True

If it has a vector, you can retrieve it from the vector attribute.

hat.vector

array([ 0.25681  , -0.35552  , -0.18733  , -0.16592  , -0.68094  ,
        0.60802  ,  0.16501  ,  0.17907  ,  0.17855  ,  1.2894   ,
       -0.46481  , -0.22667  ,  0.035198 , -0.45087  ,  0.71845  ,
                ...
       -0.94376  , -0.10265  ,  0.4415   ,  0.37775  , -0.24274  ,
       -0.42695  ,  0.18544  ,  0.16044  , -0.63395  , -0.074032 ,
       -0.038969 ,  0.30813  , -0.069243 ,  0.13493  ,  0.37585  ],
      dtype=float32)

The full vector has 300 dimensions.

A vector for a sentence is similar and has the same shape.

sent = nlp("He wore a red shirt with gray pants.").vector

array([ 8.16512257e-02, -8.81854445e-02, -1.21790558e-01, -7.65599236e-02,
        8.34635943e-02,  5.33326678e-02, -1.63263362e-02, -3.44585180e-01,
       -1.27936229e-01,  1.74646115e+00, -1.88558996e-01,  6.99177757e-02,
                ...
        1.32453769e-01, -1.40210897e-01, -5.84307760e-02,  3.93804982e-02,
        1.89477772e-01, -1.38648778e-01, -1.60174996e-01,  2.84267794e-02,
        2.16686666e-01,  1.05772227e-01,  1.48718446e-01,  9.56766680e-02],
      dtype=float32)

The sentence vector is the same shape as the word vector because it is made up of the average of the word vectors over each word in the sentence.

Formatting the input data for Scikit-learn

Ultimately the goal is to turn a list of text samples into a feature matrix, where there is a row for each text sample, and a column for each feature.

A word vector is initially a 1 x 300 column, but we want to transform it into a 300 x 1 row.

So the first step is to reshape the word vector.

sent = sent.reshape(1,-1)
sent.shape

(300,)

Then the rows are all concatenated together to create the full feature matrix.

Let’s look at an example

Say you have a corpus like the one below, with the goal of classifying the sentences as either talking about some item of clothing or not.

corpus = [
"I went outside yesterday and picked some flowers.",
"She wore a red hat with a dress to the party.", 
"I think he was wearing athletic clothes and sneakers of some sort.", 
"I took my dog for a walk at the park.", 
"I found a hot pink hat on sale over the weekend.",
"The dog has brown fur with white spots."
]

labels = [0,1,1,0,1,0]

Training labels — two classes

• 0 if not talking about clothing.
• 1 if talking about clothing.

Turning the data into a feature matrix

1. Get the vector of each sentence from Spacy.
2. Reshape each vector.
3. Concatenating the sentence vectors all together with numpy.concatenate.

import numpy as np

data_list = [nlp(doc).vector.reshape(1,-1) for doc in corpus]
data = np.concatenate(data_list)

array([[ 0.08162278,  0.15696655, -0.32472467, ...,  0.01618122,
         0.01810523,  0.2212121 ],
       [ 0.1315948 , -0.0819225 , -0.08803785, ..., -0.01854067,
         0.09653309,  0.1096675 ],
       [ 0.07139538,  0.09503647, -0.14292692, ...,  0.01818248,
         0.10714766,  0.07863422],
       [ 0.14246173,  0.18372808, -0.18847175, ...,  0.174818  ,
        -0.07943812,  0.20305632],
       [ 0.08148216,  0.09574908, -0.13909541, ..., -0.10646044,
        -0.03817916,  0.22827934],
       [-0.09829144, -0.02671766, -0.07231866, ..., -0.00786566,
         0.00078378,  0.12298879]], dtype=float32)

At this point the data is in the correct input format for many Scikit-learn algorithms.

Now we will package up this code into a reusable class that can be used in a pipeline.

Writing a Scikit-learn transformer class

We can write a custom transformer class to be used just as Scikit-learn’s TfidfVectorizer or CountVectorizer that we saw earlier.

WordVectorTransformer class

import numpy as np
import spacy
from sklearn.base import BaseEstimator, TransformerMixin

class WordVectorTransformer(TransformerMixin,BaseEstimator):
    def __init__(self, model="en_core_web_lg"):
        self.model = model

    def fit(self,X,y=None):
        return self

    def transform(self,X):
        nlp = spacy.load(self.model)
        return np.concatenate([nlp(doc).vector.reshape(1,-1) for doc in X])

• The class inherits from a couple of Scikit-learn base classes, which you can read about here in the docs.
• It needs a fit and a transform method.

This transformer initializes the Spacy model that we’re using, and then I have pretty much copied and pasted the code from earlier to create the feature matrix from the raw text samples.

One important thing to keep in mind, is that the parameters that you pass to __init__ should not be altered or changed.

In this case, I just passed the name of the Spacy model to be used, en_core_web_lg, and then the model is actually loaded in thetransform method.

At first (before reading the docs more thoroughly…) I tried to load the model in __init__ and assigned that to self.model, but that won’t work if you are using GridSearchCV with multiprocessing.

This has to do with cloning.

You can read the coding guidelines to properly build Scikit-learn components here.

So now the transformer is ready to use.

transformer = WordVectorTransformer()
transformer.fit_transform(corpus)

Using the WordVectorTransformer in a Scikit-learn Pipeline

The transformer can also be used in a pipeline.

text_clf = Pipeline([
            ('vect', WordVectorTransformer()),
            ('clf', SGDClassifier()),
            ])

This is the exact same pipeline as we saw earlier in the post, only with WordVectorTransformer instead of TfidfVectorizer.

text_clf.fit(corpus,labels)

Then call .fit() with the data samples and labels, and otherwise go about your training and testing process as usual.

Thanks for reading!

Let me know if you have questions or comments!

Write them below or feel free to reach out on Twitter @LVNGD.