Web page to excel vba

Microsoft Excel is undoubtedly one of the most used software applications around the world and across various disciplines. It not only stores, organizes, and manipulates the data using different functions and formulas but also allows users to access web pages and extract data from them.

In this tutorial, we are going to focus on the last mentioned feature by demonstrating how to perform Excel web scraping using VBA. We will briefly go through the installation and preparation of the environment and then write a scraper using VBA macro to successfully fetch data from a web page into Excel. Let’s get started.

What is VBA web scraping?

VBA web scraping is a special scraping technique that allows for automatic data gathering from websites to Excel. The scraping itself becomes possible with the use of such external applications like Microsoft Edge browser.

Pros and cons of using VBA for scraping

Before we move on to the tutorial part it is essential to highlight some advantages and disadvantages of web scraping to Excel with VBA.

Pros

• Ready to use – VBA is bundled with Microsoft Office which basically means that if you already have MS Office installed, you don’t have to worry about installing anything else. You can use VBA right away in all the Microsoft Office tools.

• Reliable – Both Microsoft Excel & VBA are developed and maintained by Microsoft. Unlike other development environments, these tools can be upgraded together to the latest version without much hassle.

• Out-of-the-box support for browser – VBA web scrapers can take advantage of Microsoft’s latest browser Microsoft Edge which makes scraping dynamic websites pretty convenient.  

• Complete automation – When running the VBA script, you don’t have to perform any additional tasks or interact with the browser. Everything will be taken care of by the VBA script including log-in, scrolling, button clicks, etc.

Cons

• Only works in Windows – VBA scrapers are not cross-platform. They only work in a Windows environment. While MS Office does have support for Mac, it is way harder to write a working VBA scraper on it. The library supports are also limited, for example, you will be unable to use Microsoft Edge.

• Tightly coupled with MS Office – VBA scrapers are highly dependent on MS Office tools. Third-party useful scraping tools are hard to integrate with it.

• Steep learning curve — VBA programming language is less beginner-friendly and a bit harder than other modern programming languages, such as Python or Javascript.

Overall, if you are looking to develop a web scraper for the Windows operating system that automatically pulls data from a website, then VBA-based web scraping will be a good choice.

Tutorial

Before we begin, let us make sure we’ve installed all the prerequisites and set up our environment properly so that it will be easier to follow along.

Prerequisites

In this tutorial, we’ll be using Windows 10 and Microsoft Office 10. However, the steps will be the same or similar for other versions of Windows. You’ll only need a computer with Windows Operating System. In addition, it’s necessary to install Microsoft Office if you don’t have it already. Detailed installation instructions can be found in the Microsoft’s Official documentation.

Preparing the environment

Now, that you’ve installed MS Office, complete the steps below to set up the development environment:

Step 1 — Open Microsoft Excel

From the start menu or Cortana search, find Microsoft Excel and open the application. You will see a similar interface as below: 

Click on File 

Step 2 — Go to Options to enable developer menu

By default, Excel doesn’t show the developer button in the top ribbon. To enable this, we’ll have to go to “Options” from the File menu.

Step 3 —  Select Customize Ribbon

Once you click the “Options,” a dialog will pop up where, from the side menu, you’ll need to select “Customize Ribbon”. Click on the check box next to “developer.” Make sure it is ticked and then click on “OK.”

Step 4 — Open Visual Basic application dialog

Now, you’ll see a new developer button on the top ribbon, clicking on it will expand the developer menu. From the menu, select “Visual Basic.”

Step 5 — Insert a new Module

Once you click on “Visual Basic», it will open a new window as demonstrated below:

Click on “Insert” and select “Module” to insert a new module. It will open the module editor.

Step 6 — Add new references

From the top menu select Tools >  References… which will open a new window like the one below. Make sure to scroll through the available list of references and find Microsoft HTML Client Library and Microsoft Internet Control. Click on the check box next to both of them to enable these references. Once done, click OK.

That’s it! Our development environment is all set. Let’s write our first Excel VBA scraper.

Step 7 — Automate Microsoft Edge to open a website

Now, it’s time to update our newly created module to open the following website: https://quotes.toscrape.com. In the module editor, insert the following code:

Sub scrape_quotes()
   Dim browser As InternetExplorer
   Dim page As HTMLDocument
   Set browser = New InternetExplorer
   browser.Visible = True
   browser.navigate ("https://quotes.toscrape.com")
End Sub

We are defining a subroutine named scrape_quotes(). This function will be executed when we run this script. Inside the subroutine, we are defining two objects “browser” and  “page”. 

The “browser” object will allow us to interact with Microsoft Edge. Next, we also set the browser as visible so that we can see it in action. The browser.navigate() function tells the VBA browser object to open the URL. The output will be similar to this:

Note: You might be wondering why we are writing “InternetExplorer” to interact with Microsoft Edge. VBA initially only supported Internet Explorer-based automation, but once Microsoft discontinued Internet Explorer, they deployed some updates so that VBA’s InternetExplorer module can run the Microsoft Edge browser in IEMode without any issues. The above code will also work in older Windows that have Internet Explorer still available instead of Edge. 

Step 8 — Scrape data using VBA script and save it to Excel

The next step is to scrape the quotes and authors from the website. For simplicity, we’ll store it in the first sheet of the Excel spreadsheet and grab the top 5 quotes for now.

Let’s begin by defining two new objects – one for quotes and another for authors.

   Dim quotes As Object
   Dim authors As Object

After navigating to the website, we’ll also add a little bit of pause so that the website loads properly by using Loop.

   Do While browser.Busy: Loop

Next, grab the quotes and authors from the HTML document.

   Set page = browser.document
   Set quotes = page.getElementsByClassName("quote")
   Set authors = page.getElementsByClassName("author")

Use a for loop to populate the excel rows with the extracted data by calling the Cells function and passing the row and column position:

   For num = 1 To 5
       Cells(num, 1).Value = quotes.Item(num).innerText
       Cells(num, 2).Value = authors.Item(num).innerText
   Next num

Finally, close the browser by calling the quit function. The below code will close the browser window.

Output

Now, if we run the script again, it’ll open Microsoft Edge, browse to the quotes.toscrape.com website, grab the top 5 quotes from the list, and save them to the current excel file’s first sheet.

Source Code

Below is an example of a full source code:

Sub scrape_quotes()
   Dim browser As InternetExplorer
   Dim page As HTMLDocument
   Dim quotes As Object
   Dim authors As Object
  
   Set browser = New InternetExplorer
   browser.Visible = True
   browser.navigate ("https://quotes.toscrape.com")
   Do While browser.Busy: Loop
  
   Set page = browser.document
   Set quotes = page.getElementsByClassName("quote")
   Set authors = page.getElementsByClassName("author")
  
   For num = 1 To 5
       Cells(num, 1).Value = quotes.Item(num).innerText
       Cells(num, 2).Value = authors.Item(num).innerText
   Next num
  
   browser.Quit
End Sub

Conclusion

Excel web scraping with VBA is a great choice for Windows automation and web extraction. It’s reliable and ready to use which means you won’t have to worry about any unexpected issues or additional steps. For your convenience, you can also access this tutorial in our GitHub repository. 

The biggest disadvantage of VBA web scraping that was highlighted in the article is the lack of cross-platform support. However, if you want to develop web scrapers that can be used on multiple operating systems such as Linux or Mac, Excel Web Query can also be an option. Of course, we also recommend exploring web scraping with Python – one of the most popular programming languages that is capable of developing complex network applications while maintaining its simplified syntax. 

About the author

Yelyzaveta Nechytailo

Senior Content Manager

Yelyzaveta Nechytailo is a Senior Content Manager at Oxylabs. After working as a writer in fashion, e-commerce, and media, she decided to switch her career path and immerse in the fascinating world of tech. And believe it or not, she absolutely loves it! On weekends, you’ll probably find Yelyzaveta enjoying a cup of matcha at a cozy coffee shop, scrolling through social media, or binge-watching investigative TV series.

All information on Oxylabs Blog is provided on an «as is» basis and for informational purposes only. We make no representation and disclaim all liability with respect to your use of any information contained on Oxylabs Blog or any third-party websites that may be linked therein. Before engaging in scraping activities of any kind you should consult your legal advisors and carefully read the particular website’s terms of service or receive a scraping license.

Related articles

VBA Code to Extract Data from Website to Excel

To perform HTML screen scrapping or to automatically pull data from website into Excel vba, use this simple code in this page.

Programmers & Software testing Professionals use this technique when ‘Excel query’ to fetch webpage option could not get required data from web or due to data formatting issues. For example, if we need a stock quotes in Excel, then we can use a Excel webquery option. But, we cannot hand pick the required data alone.

Lets see how to pull data from a website with Excel VBA code automatically.

Also Read: How to automatically pull website data from Chrome using Excel VBA & Selenium Type library?

How to automatically pull data from Website to Excel using VBA?

HTML screen scraping can be done by creating instances of the object “MSXML2.XMLHTTP” or “MSXML2.ServerXMLHTTP”, as explained in the code snippet below.

Use this code in VBA to pull data from website i.,e HTML source of a page & place the content in a excel sheet.

1. Create new excel workbook
2. Press Alt + F11 to open VB editor.
3. Copy paste the below code in space provided for code.
4. Change the URL mentioned in the code.
5. Execute the code by Pressing F5.

Private Sub HTML_VBA_Pull_Data_From_Website_To_Excel()
    Dim oXMLHTTP As Object
    Dim sPageHTML  As String
    Dim sURL As String

    'Change the URL before executing the code. URL to Extract data from.
    sURL = "http://WWW.WebSiteName.com"

    'Pull/Extract data from website to Excel using VBA
    Set oXMLHTTP = CreateObject("MSXML2.XMLHTTP")
    oXMLHTTP.Open "GET", sURL, False
    oXMLHTTP.send
    sPageHTML = oXMLHTTP.responseText

    'Get webpage data into Excel
    ThisWorkbook.Sheets(1).Cells(1, 1) = sPageHTML

    MsgBox "XMLHTML Fetch Completed"

End Sub

Once you run the above code, it will automatically pull website data in Excel vba as a HTML code.

You can parse the HTML elements from the web page as explained in out another article: How To Extract HTML Table From Website?

It explains with a same VBA HTML Parser code on how to process each HTML element, after the web page content is extracted.

Clear Cache for every VBA Extract Data from Website

Use this below code every time, before executing the above code. Because with every execution, it is possible that extracted website data can reside in cache.

Also Read: Extract URL From Sitemap of a Website

If you run the code again without clearing the cache, then old data will be displayed again. To avoid this, use the below code.

Shell "RunDll32.exe InetCpl.Cpl, ClearMyTracksByProcess 11"

Note: Once this code is executed, it will clear the cache in your web browser. So, please execute this with caution. All your previous sessions and unfinished work will be deleted from browser cache.

Detect Broken URL or Dead Links In Webpage

Sometimes we might need to check only if the URL is active or not. During that time, just insert the below checking IF condition just after the .Send command.

If the URL is not active, then this code snippet will throw an error message.

    oXMLHTTP.send

    If oXMLHTTP.Status <> 200 Then
        MsgBox sURL & ": URL Link is not Active"
    End If

Search Engines Extract Website Data

Web browsers like IE, Chrome, Firefox etc., fetch data from webpages and presents us in beautiful formats.

Why do we need to download Website data with a program? Even few crazy programmers develop such applications & name it as Spider or Crawler? Because it is going to Crawl through the web and extract HTML content from different websites.

Internet Search Engines: Google, Bing etc., have crawler programs that download webpage content and index. Guessing, nobody is trying to  build a search engine with Excel, but may be for processing website content for other purposes like counting for words, checking specific URLs, Descriptions or Correctness of Email IDs etc.,

Onpage SEO Analysis: Web admins want to fetch their own website data for SEO analysis, marketing people do it for extracting email ids for promoting their products.

Excel Query Fails: Some websites get updated so frequently, like Stock Index, Facebook timeline, Sports updates, Commodity prices, News updates etc.,In this case one would not like to click on ‘Refresh Page’ multiple times.

Use this VBA to extract data from website and accomplish Automated content processing.

Additional References

Though this article explained about fetching webpage content, the options available with MSXML2.ServerXMLHTTP are not discussed. To know more about this options, visit to MSDN Library.

This blog shows how to go through a website, making sense of its HTML within
VBA.  We’ll break the problem down into several chunks — to whit: 

1. Defining what we want to achieve.
2. Analysing the target URL (the target website).
3. Referencing the required applications.
4. Getting at the underlying HTML.
5. Parsing the data into Excel.

Don’t forget that websites change all the time, so this code may no longer
work when you try it out as the format of the
StackOverflow website may have changed. 
The following code also assumes that you have Internet Explorer on your computer
(something which will be true of nearly all Windows computers).

Step 1 — Defining the problem

At the time of writing, here is what the above-mentioned StackOverflow
website’s home page looks like:

From this we want to extract the raw questions, with just the votes, views and
author information:

To do this we need to learn the structure of the HTML behind the page.

To scrape websites you need to know a little HTML, and knowing a lot will
help you enormously.

Step 2 — Analysing the target website

In any browser you can right-click and choose to show the underlying HTML for
the page:

The web page begins with Top Questions, so let’s find that:

Analysing the HTML which follows this shows that the questions are all encased
in a div tag called question-mini-list:

Here’s the HTML for a single question:

Here’s how we’ll get at the four bits of data we want:

Step 3 — Referencing the required applications

To get this macro to work, we’ll need to:

• Create a new copy of Internet Explorer in memory; then
• Work with the elements on the HTML page we find.

To do this, you’ll need to reference two object libraries:

To do this, in VBA choose from the menu Tools —>
References, then tick the two options shown:

Now we can begin writing the VBA to get at our data!

Step 4 — Getting at the underlying HTML

Let’s now show some code for loading up the HTML at a given web page. 
The main problem is that we have to wait until the web browser has responded, so
we keep «doing any events» until it returns the correct state out of the
following choices:

Here a subroutine to get at the text behind a web page:

What this does is:

1. Creates a new copy of Internet Explorer to run invisibly in
   memory.
2. Navigates to the StackOverflow home page.
3. Waits until the home page has loaded.
4. Loads up an HTML document, and shows its text.
5. Closes Internet Explorer.

You could now parse the HTML using the Document Object Model (for those who
know this), but we’re going to do it the slightly harder way, by finding tags
and then looping over their contents. 

Step 5 — Parsing the HTML

Here’s the entire subroutine, in parts, with comments for the HTML bits. 
Start by getting a handle on the HTML document, as above:

Now put titles in row 3 of the spreadsheet:

We’re going to need a fair few variables (I don’t guarantee that this is the
most efficient solution!):

Start by getting a reference to the HTML element which contains all of the
questions (this also initialises the row number in the spreadsheet to 4, the one
after the titles):

Now we’ll loop over all of the child elements within this tag, finding each
question in turn:

Each question has a tag giving its id, which we can extract:

Now we’ll loop over all of the child elements within each question’s
containing div tag:

For each element, extract its details (either the integer number of votes
cast, the integer number of views or the name of the author):

Time now to finish this question, increase the spreadsheet row count by one
and go on to the next question:

Finally, we’ll tidy up the results and put a title in row one:

And that’s the complete macro! 

As the above shows, website scraping can get quite messy.  If you’re
going to be doing much of this, I recommend learning about the HTML DOM
(Document Object Model), and taking advantage of this in your code.

If you’ve learnt something from this blog and wonder how much more we could
teach you, have a look at our online and classroom VBA courses.