To Parse Custom JSON data is to split out its name/value pairs into a more readable useable format.
Excel is a powerful tool that allows you to connect to JSON data and read it. However sometimes this data might require a little manipulation to be fully understood and analysed in Excel.
In this article you will learn
- What is JSON Data
- To Connect to JSON data from Excel
- How to Parse simple JSON Data using Excels Power Query
- To Parse complex JSON Data using Excels Power Query
This article contains data tables to download so you can practice along and master the art of parsing custom JSON data using Excel. We are also powered with STEEM so you can earn while you learn . If you are not familiar with Excel Power Tools you can find out about them here.
What is JSON Data?
JSON data is a way of representing objects or arrays. It is easy to read, and it is easy to parse, even with Excel. Many API calls will return JSON format and many web apps use JSON which easily moves information around the internet.
The syntax for JSON comes from JavaScript and can be summarized as follows:
- Data is shown in name/value pairs separated by : For example “name” : ”paulag”
- Data is separated by commas. For Example “name” : ”paulag” , “Sex” : “Female”
- Curly brackets represent an object. An object being a name/value pair separated by a comma. For Example {“name” : ”paulag” , “Sex” : “Female”}
- Square brackets hold arrays and contains a list of values separated by a comma.
Look at the sample JSON in the image below
{“total_population”: This shows the first object, which is a name/value pair. The name of the object is total_population
[{“date”: “2019-01-02”, “population”: 7638335801}, {“date”: “2019-01-03”, “population”: 7638557789}]} This is the value for the total population. The [ represents an array. This array contains two objects. The objects are defined within the curly brackets and separated with a comma. Each object contains 2 lots of data (name/value pair) also separated with a comma. The data, shown in the name/value pairs, in this example is date and population.
Here is one you can look at yourself. https://api.coinmarketcap.com/v1/ticker/bitcoin/
If you enter this to your browser, you will get something like the below:
Let’s take a look at that image in more detail.
Connecting to JSON data from Excel
In Excels Data ribbon, under GET and Transform Data, we have the option of connecting to data of multiple sources and multiple types.
If we select Get data from file, we will then have the option to get data from a JSON file.
In this example we have URL API endpoint https://api.coinmarketcap.com/v1/ticker/bitcoin/. Therefore, from the Data Ribbon we can select, Get data from Web. This will open a dialogue box in which you place the URL.
Next, Power query will then open. Power query is a magic excel tool that will allow you transform data that you connect to into a usable format.
The JSON data will appear as a list of records Power Query. For excel to read this, we must convert a list to a table. Select ‘to table’ from the available option.
Next, Power query will create a table and you will see this step appear on the right of the power query window under applied steps.
This new table contains a record. We must expand this record to get the value pairs. As this record only has 1 row, we would expect this to expand across the columns. To do this, right click on the arrows in the column header.
This will reveal the names of the value pairs. By selecting ok a new column will be set up in the table. The name will be in the header and the value in the row.
Further transformations
If we wanted this data going down the row, we could Unpivot the columns. By selecting the id column. Then from the Transform ribbon select the dropdown for unpivot columns and select unpivot other columns.
When working with Power Query, it’s important to make sure you have the correct data types set. To work with this data, we must now move to from Power Query to Excel. If we select File, and then select Close and load, this will load the data as a table in Excel. Or, if we select or Close and Load to, the data will be loaded into a Power Pivot Model.
How to Parse JSON Data in Excel
Very often when you access JSON data with Excel it appears in 1 column. This can happen for many reasons and is often the design of a database.
Look at the image below. We can see the json_metadata field is still in its JSON syntax
When we encounter data like this, we can easily parse the column into its components. From the image below we see we have 4 components. We have An Array, an Object, the data, and one of the data fields contains an array.
STEP by STEP
Download this file. It contains a table as shown below. (do not copy and paste the table as the JSON field will not be recognised.)
tx_id |
tid |
json_metadata |
timestamp |
647524676 |
follow |
[“follow”,{“follower”:”mervin-gil”,”following”:”jarvie”,”what”:[“blog”]}] |
43466.89097 |
647524682 |
follow |
[“follow”,{“follower”:”steliosfan”,”following”:”michealb”,”what”:[“blog”]}] |
43466.89097 |
647524833 |
follow |
[“follow”,{“follower”:”eugenezh”,”following”:”zentricbau”,”what”:[“blog”]}] |
43466.89097 |
647524855 |
follow |
[“follow”,{“follower”:”bitcoinportugal”,”following”:”manuellevi”,”what”:[]}] |
43466.89097 |
647525074 |
follow |
[“follow”,{“follower”:”eugenezh”,”following”:”adriellute”,”what”:[“blog”]}] |
43466.89167 |
647525089 |
follow |
[“follow”,{“follower”:”bigbigtoe”,”following”:”hoxly”,”what”:[“blog”]}] |
43466.89167 |
647525121 |
follow |
[“follow”,{“follower”:”mervin-gil”,”following”:”loveon”,”what”:[“blog”]}] |
43466.89167 |
647525159 |
follow |
[“follow”,{“follower”:”mervin-gil”,”following”:”pechichemena”,”what”:[“blog”]}] |
43466.89167 |
647525233 |
follow |
[“follow”,{“follower”:”imealien”,”following”:”pataty69″,”what”:[“blog”]}] |
43466.89167 |
647525652 |
follow |
[“follow”,{“follower”:”mervin-gil”,”following”:”kamile”,”what”:[“blog”]}] |
43466.89236 |
647525818 |
follow |
[“follow”,{“follower”:”bitcoinportugal”,”following”:”drmake”,”what”:[“blog”]}] |
43466.89236 |
647525886 |
follow |
[“follow”,{“follower”:”mervin-gil”,”following”:”bradfordtennyson”,”what”:[“blog”]}] |
43466.89236 |
647525980 |
follow |
[“follow”,{“follower”:”a0i”,”following”:”shoemanchu”,”what”:[“blog”]}] |
43466.89236 |
647526007 |
follow |
[“follow”,{“follower”:”voteme”,”following”:”kostyantin”,”what”:[“blog”]}] |
43466.89236 |
648215552 |
follow |
[“follow”,{“follower”:”ansie”,”following”:”hoxly”,”what”:[“blog”]}] |
43467.75833 |
648215582 |
follow |
[“follow”,{“follower”:”ashokcan143″,”following”:”abcor”,”what”:[]}] |
43467.75833 |
648215691 |
follow |
[“follow”,{“follower”:”ashokcan143″,”following”:”abasinkanga”,”what”:[]}] |
43467.75903 |
648215820 |
follow |
[“follow”,{“follower”:”nongvo.piy”,”following”:”acidyo”,”what”:[]}] |
43467.75903 |
648215859 |
follow |
[“follow”,{“follower”:”grid9games”,”following”:”yeninsfer”,”what”:[“blog”]}] |
43467.75903 |
648215945 |
follow |
[“follow”,{“follower”:”nongvo.piy”,”following”:”acidyo”,”what”:[“blog”]}] |
43467.75903 |
Click on any of the cells that contain the data and from the Data Ribbon select FROM Table/Range
If your data is not in table format, Excel will then prompt you to create a table. Select the cells the contain the data and tick the box to say that your table has headers.
Power Query editor window will open. On the canvas you can see your data and, on the right, you can see any transformation steps that have taken place.
To parse the json column, first select the column, then on the Transform Ribbon select Parse and select JSON
Power query will recognise the first [ and create a list format. Next, we need to expand this list to new rows. To do this click on the arrows on the top of the column and select Expand to New Rows.
What is returned is two lines for each tx-id. The json column now has a row for the name of the array, which is follow, and a record. The record will contain the data.
As we do not need the name of the array, we can use the filter to remove all the follow rows
We are now left with the records. We can expand this record, by pressing the arrow on the column. From here we see we have the names of 3 data fields, Follower, Following and What
When we select OK, we get a new column in our data table for each data field.
However, the field named what contains an array, which is again shown as a list (or array) which needs to be expanded
Once we expand this, we are at the end of the JSON data and have extracted the relevant columns. You can now use this data for further analysis in Excel or Power Pivot. To load it back to excel, select File and Close & Load.
More complex JSON data extractions in Excel
So far, we have looked at getting JSON data into Power Query using an Excel table and directly from a URL. There are other ways you can connect to JSON data including connecting directly to a JSON file. Connecting is the easy part. Things get more complicated when you have JSON columns where the strings are different in each row. One might start with an array and so return a list, but some might start with an object and return a record.
In Power Query lists are expanded to new rows in the table and records are expanded to new columns. Parsing custom JSON data in Excel can require some thinking.
UPDATE 3 (Sep 24 ’17)
Check VBA-JSON-parser on GitHub for the latest version and examples. Import JSON.bas module into the VBA project for JSON processing.
UPDATE 2 (Oct 1 ’16)
However if you do want to parse JSON on 64-bit Office with ScriptControl
, then this answer may help you to get ScriptControl
to work on 64-bit.
UPDATE (Oct 26 ’15)
Note that a ScriptControl
-based approachs makes the system vulnerable in some cases, since they allows a direct access to the drives (and other stuff) for the malicious JS code via ActiveX’s. Let’s suppose you are parsing web server response JSON, like JsonString = "{a:(function(){(new ActiveXObject('Scripting.FileSystemObject')).CreateTextFile('C:\Test.txt')})()}"
. After evaluating it you’ll find new created file C:Test.txt
. So JSON parsing with ScriptControl
ActiveX is not a good idea.
Trying to avoid that, I’ve created JSON parser based on RegEx’s. Objects {}
are represented by dictionaries, that makes possible to use dictionary’s properties and methods: .Count
, .Exists()
, .Item()
, .Items
, .Keys
. Arrays []
are the conventional zero-based VB arrays, so UBound()
shows the number of elements. Here is the code with some usage examples:
Option Explicit
Sub JsonTest()
Dim strJsonString As String
Dim varJson As Variant
Dim strState As String
Dim varItem As Variant
' parse JSON string to object
' root element can be the object {} or the array []
strJsonString = "{""a"":[{}, 0, ""value"", [{""stuff"":""content""}]], b:null}"
ParseJson strJsonString, varJson, strState
' checking the structure step by step
Select Case False ' if any of the checks is False, the sequence is interrupted
Case IsObject(varJson) ' if root JSON element is object {},
Case varJson.Exists("a") ' having property a,
Case IsArray(varJson("a")) ' which is array,
Case UBound(varJson("a")) >= 3 ' having not less than 4 elements,
Case IsArray(varJson("a")(3)) ' where forth element is array,
Case UBound(varJson("a")(3)) = 0 ' having the only element,
Case IsObject(varJson("a")(3)(0)) ' which is object,
Case varJson("a")(3)(0).Exists("stuff") ' having property stuff,
Case Else
MsgBox "Check the structure step by step" & vbCrLf & varJson("a")(3)(0)("stuff") ' then show the value of the last one property.
End Select
' direct access to the property if sure of structure
MsgBox "Direct access to the property" & vbCrLf & varJson.Item("a")(3)(0).Item("stuff") ' content
' traversing each element in array
For Each varItem In varJson("a")
' show the structure of the element
MsgBox "The structure of the element:" & vbCrLf & BeautifyJson(varItem)
Next
' show the full structure starting from root element
MsgBox "The full structure starting from root element:" & vbCrLf & BeautifyJson(varJson)
End Sub
Sub BeautifyTest()
' put sourse JSON string to "desktopsource.json" file
' processed JSON will be saved to "desktopresult.json" file
Dim strDesktop As String
Dim strJsonString As String
Dim varJson As Variant
Dim strState As String
Dim strResult As String
Dim lngIndent As Long
strDesktop = CreateObject("WScript.Shell").SpecialFolders.Item("Desktop")
strJsonString = ReadTextFile(strDesktop & "source.json", -2)
ParseJson strJsonString, varJson, strState
If strState <> "Error" Then
strResult = BeautifyJson(varJson)
WriteTextFile strResult, strDesktop & "result.json", -1
End If
CreateObject("WScript.Shell").PopUp strState, 1, , 64
End Sub
Sub ParseJson(ByVal strContent As String, varJson As Variant, strState As String)
' strContent - source JSON string
' varJson - created object or array to be returned as result
' strState - Object|Array|Error depending on processing to be returned as state
Dim objTokens As Object
Dim objRegEx As Object
Dim bMatched As Boolean
Set objTokens = CreateObject("Scripting.Dictionary")
Set objRegEx = CreateObject("VBScript.RegExp")
With objRegEx
' specification http://www.json.org/
.Global = True
.MultiLine = True
.IgnoreCase = True
.Pattern = """(?:\""|[^""])*""(?=s*(?:,|:|]|}))"
Tokenize objTokens, objRegEx, strContent, bMatched, "str"
.Pattern = "(?:[+-])?(?:d+.d*|.d+|d+)e(?:[+-])?d+(?=s*(?:,|]|}))"
Tokenize objTokens, objRegEx, strContent, bMatched, "num"
.Pattern = "(?:[+-])?(?:d+.d*|.d+|d+)(?=s*(?:,|]|}))"
Tokenize objTokens, objRegEx, strContent, bMatched, "num"
.Pattern = "b(?:true|false|null)(?=s*(?:,|]|}))"
Tokenize objTokens, objRegEx, strContent, bMatched, "cst"
.Pattern = "b[A-Za-z_]w*(?=s*:)" ' unspecified name without quotes
Tokenize objTokens, objRegEx, strContent, bMatched, "nam"
.Pattern = "s"
strContent = .Replace(strContent, "")
.MultiLine = False
Do
bMatched = False
.Pattern = "<d+(?:str|nam)>:<d+(?:str|num|obj|arr|cst)>"
Tokenize objTokens, objRegEx, strContent, bMatched, "prp"
.Pattern = "{(?:<d+prp>(?:,<d+prp>)*)?}"
Tokenize objTokens, objRegEx, strContent, bMatched, "obj"
.Pattern = "[(?:<d+(?:str|num|obj|arr|cst)>(?:,<d+(?:str|num|obj|arr|cst)>)*)?]"
Tokenize objTokens, objRegEx, strContent, bMatched, "arr"
Loop While bMatched
.Pattern = "^<d+(?:obj|arr)>$" ' unspecified top level array
If Not (.Test(strContent) And objTokens.Exists(strContent)) Then
varJson = Null
strState = "Error"
Else
Retrieve objTokens, objRegEx, strContent, varJson
strState = IIf(IsObject(varJson), "Object", "Array")
End If
End With
End Sub
Sub Tokenize(objTokens, objRegEx, strContent, bMatched, strType)
Dim strKey As String
Dim strRes As String
Dim lngCopyIndex As Long
Dim objMatch As Object
strRes = ""
lngCopyIndex = 1
With objRegEx
For Each objMatch In .Execute(strContent)
strKey = "<" & objTokens.Count & strType & ">"
bMatched = True
With objMatch
objTokens(strKey) = .Value
strRes = strRes & Mid(strContent, lngCopyIndex, .FirstIndex - lngCopyIndex + 1) & strKey
lngCopyIndex = .FirstIndex + .Length + 1
End With
Next
strContent = strRes & Mid(strContent, lngCopyIndex, Len(strContent) - lngCopyIndex + 1)
End With
End Sub
Sub Retrieve(objTokens, objRegEx, strTokenKey, varTransfer)
Dim strContent As String
Dim strType As String
Dim objMatches As Object
Dim objMatch As Object
Dim strName As String
Dim varValue As Variant
Dim objArrayElts As Object
strType = Left(Right(strTokenKey, 4), 3)
strContent = objTokens(strTokenKey)
With objRegEx
.Global = True
Select Case strType
Case "obj"
.Pattern = "<d+w{3}>"
Set objMatches = .Execute(strContent)
Set varTransfer = CreateObject("Scripting.Dictionary")
For Each objMatch In objMatches
Retrieve objTokens, objRegEx, objMatch.Value, varTransfer
Next
Case "prp"
.Pattern = "<d+w{3}>"
Set objMatches = .Execute(strContent)
Retrieve objTokens, objRegEx, objMatches(0).Value, strName
Retrieve objTokens, objRegEx, objMatches(1).Value, varValue
If IsObject(varValue) Then
Set varTransfer(strName) = varValue
Else
varTransfer(strName) = varValue
End If
Case "arr"
.Pattern = "<d+w{3}>"
Set objMatches = .Execute(strContent)
Set objArrayElts = CreateObject("Scripting.Dictionary")
For Each objMatch In objMatches
Retrieve objTokens, objRegEx, objMatch.Value, varValue
If IsObject(varValue) Then
Set objArrayElts(objArrayElts.Count) = varValue
Else
objArrayElts(objArrayElts.Count) = varValue
End If
varTransfer = objArrayElts.Items
Next
Case "nam"
varTransfer = strContent
Case "str"
varTransfer = Mid(strContent, 2, Len(strContent) - 2)
varTransfer = Replace(varTransfer, """", """")
varTransfer = Replace(varTransfer, "\", "")
varTransfer = Replace(varTransfer, "/", "/")
varTransfer = Replace(varTransfer, "b", Chr(8))
varTransfer = Replace(varTransfer, "f", Chr(12))
varTransfer = Replace(varTransfer, "n", vbLf)
varTransfer = Replace(varTransfer, "r", vbCr)
varTransfer = Replace(varTransfer, "t", vbTab)
.Global = False
.Pattern = "\u[0-9a-fA-F]{4}"
Do While .Test(varTransfer)
varTransfer = .Replace(varTransfer, ChrW(("&H" & Right(.Execute(varTransfer)(0).Value, 4)) * 1))
Loop
Case "num"
varTransfer = Evaluate(strContent)
Case "cst"
Select Case LCase(strContent)
Case "true"
varTransfer = True
Case "false"
varTransfer = False
Case "null"
varTransfer = Null
End Select
End Select
End With
End Sub
Function BeautifyJson(varJson As Variant) As String
Dim strResult As String
Dim lngIndent As Long
BeautifyJson = ""
lngIndent = 0
BeautyTraverse BeautifyJson, lngIndent, varJson, vbTab, 1
End Function
Sub BeautyTraverse(strResult As String, lngIndent As Long, varElement As Variant, strIndent As String, lngStep As Long)
Dim arrKeys() As Variant
Dim lngIndex As Long
Dim strTemp As String
Select Case VarType(varElement)
Case vbObject
If varElement.Count = 0 Then
strResult = strResult & "{}"
Else
strResult = strResult & "{" & vbCrLf
lngIndent = lngIndent + lngStep
arrKeys = varElement.Keys
For lngIndex = 0 To UBound(arrKeys)
strResult = strResult & String(lngIndent, strIndent) & """" & arrKeys(lngIndex) & """" & ": "
BeautyTraverse strResult, lngIndent, varElement(arrKeys(lngIndex)), strIndent, lngStep
If Not (lngIndex = UBound(arrKeys)) Then strResult = strResult & ","
strResult = strResult & vbCrLf
Next
lngIndent = lngIndent - lngStep
strResult = strResult & String(lngIndent, strIndent) & "}"
End If
Case Is >= vbArray
If UBound(varElement) = -1 Then
strResult = strResult & "[]"
Else
strResult = strResult & "[" & vbCrLf
lngIndent = lngIndent + lngStep
For lngIndex = 0 To UBound(varElement)
strResult = strResult & String(lngIndent, strIndent)
BeautyTraverse strResult, lngIndent, varElement(lngIndex), strIndent, lngStep
If Not (lngIndex = UBound(varElement)) Then strResult = strResult & ","
strResult = strResult & vbCrLf
Next
lngIndent = lngIndent - lngStep
strResult = strResult & String(lngIndent, strIndent) & "]"
End If
Case vbInteger, vbLong, vbSingle, vbDouble
strResult = strResult & varElement
Case vbNull
strResult = strResult & "Null"
Case vbBoolean
strResult = strResult & IIf(varElement, "True", "False")
Case Else
strTemp = Replace(varElement, """", """")
strTemp = Replace(strTemp, "", "\")
strTemp = Replace(strTemp, "/", "/")
strTemp = Replace(strTemp, Chr(8), "b")
strTemp = Replace(strTemp, Chr(12), "f")
strTemp = Replace(strTemp, vbLf, "n")
strTemp = Replace(strTemp, vbCr, "r")
strTemp = Replace(strTemp, vbTab, "t")
strResult = strResult & """" & strTemp & """"
End Select
End Sub
Function ReadTextFile(strPath As String, lngFormat As Long) As String
' lngFormat -2 - System default, -1 - Unicode, 0 - ASCII
With CreateObject("Scripting.FileSystemObject").OpenTextFile(strPath, 1, False, lngFormat)
ReadTextFile = ""
If Not .AtEndOfStream Then ReadTextFile = .ReadAll
.Close
End With
End Function
Sub WriteTextFile(strContent As String, strPath As String, lngFormat As Long)
With CreateObject("Scripting.FileSystemObject").OpenTextFile(strPath, 2, True, lngFormat)
.Write (strContent)
.Close
End With
End Sub
One more opportunity of this JSON RegEx parser is that it works on 64-bit Office, where ScriptControl isn’t available.
INITIAL (May 27 ’15)
Here is one more method to parse JSON in VBA, based on ScriptControl
ActiveX, without external libraries:
Sub JsonTest()
Dim Dict, Temp, Text, Keys, Items
' Converting JSON string to appropriate nested dictionaries structure
' Dictionaries have numeric keys for JSON Arrays, and string keys for JSON Objects
' Returns Nothing in case of any JSON syntax issues
Set Dict = GetJsonDict("{a:[[{stuff:'result'}]], b:''}")
' You can use For Each ... Next and For ... Next loops through keys and items
Keys = Dict.Keys
Items = Dict.Items
' Referring directly to the necessary property if sure, without any checks
MsgBox Dict("a")(0)(0)("stuff")
' Auxiliary DrillDown() function
' Drilling down the structure, sequentially checking if each level exists
Select Case False
Case DrillDown(Dict, "a", Temp, "")
Case DrillDown(Temp, 0, Temp, "")
Case DrillDown(Temp, 0, Temp, "")
Case DrillDown(Temp, "stuff", "", Text)
Case Else
' Structure is consistent, requested value found
MsgBox Text
End Select
End Sub
Function GetJsonDict(JsonString As String)
With CreateObject("ScriptControl")
.Language = "JScript"
.ExecuteStatement "function gettype(sample) {return {}.toString.call(sample).slice(8, -1)}"
.ExecuteStatement "function evaljson(json, er) {try {var sample = eval('(' + json + ')'); var type = gettype(sample); if(type != 'Array' && type != 'Object') {return er;} else {return getdict(sample);}} catch(e) {return er;}}"
.ExecuteStatement "function getdict(sample) {var type = gettype(sample); if(type != 'Array' && type != 'Object') return sample; var dict = new ActiveXObject('Scripting.Dictionary'); if(type == 'Array') {for(var key = 0; key < sample.length; key++) {dict.add(key, getdict(sample[key]));}} else {for(var key in sample) {dict.add(key, getdict(sample[key]));}} return dict;}"
Set GetJsonDict = .Run("evaljson", JsonString, Nothing)
End With
End Function
Function DrillDown(Source, Prop, Target, Value)
Select Case False
Case TypeName(Source) = "Dictionary"
Case Source.exists(Prop)
Case Else
Select Case True
Case TypeName(Source(Prop)) = "Dictionary"
Set Target = Source(Prop)
Value = Empty
Case IsObject(Source(Prop))
Set Value = Source(Prop)
Set Target = Nothing
Case Else
Value = Source(Prop)
Set Target = Nothing
End Select
DrillDown = True
Exit Function
End Select
DrillDown = False
End Function
One increasingly common task for Excel users is to retrieve data from the internet.
Simply, retrieving data from the web is the first challenge.
This sounds simple enough (and as you’ll soon see, it is).
But most people run into a roadblock with the second challenge: wrestling with data in JSON format.
Read on to discover the easiest and fastest method to get JSON data into Excel…
How to Import & Parse JSON Data with VBA
You’ve got several approaches to parsing data from the web which is in JSON format:
1) Roll Your Own
You could write your own VBA code to parse the JSON data.
This approach is only recommended if you have acute masochistic tendencies.
This path (of parsing JSON) has been travelled by others before you.
Leverage off their work 😉
2) Free / Open-Source Libraries
You can use free or open-source VBA libraries like VB-JSON or VBA-JSON.
While many of these libraries do work, some do have their own issues, may not be supported and often suffer from performance issues.
3) Third Party Libraries
You could buy a parsing library from a third-party provider.
But if you’re in a corporate environment, you may find it difficult to purchase non-standard software products.
In one recent project, it took 93 days from request to authorisation to installation to get my favourite VBA productivity suite, MZ-Tools installed.
Good thing I wasn’t in a hurry.
But if you are . . .
4) Standard COM components
The last approach, which we’ll cover here, is using standard COM components.
At the end of the day, Excel is just one big component.
What this means is that we can leverage other COM components and libraries quickly and (usually) easily.
Oh, and did I mention they’re free 🙂
What the heck is JSON?
Before we get cracking, let’s give a little attention to JSON.
You’re probably already familiar with XML.
Just like HTML uses tags to structure a webpage, XML uses tags to structure data. This data can be saved to a text file or transmitted over the internet.
JSON, which stands for JavaScript Object Notation, is another method of structuring data, although it doesn’t use tags.
Originally designed for JavaScript, JSON is now used widely for transmitting data between apps and across the web.
You can read all the gory details here.
Using a Web-service to Get Data
The web-service we’ll use to get data in JSON format for this example is Datamuse.
Datamuse is a word-finding query engine. It lets you retrieve a list of words based on a variety of criteria including meaning, spelling, sound and vocabulary.
For example, you could retrieve a list of words related to duck that start with the letter ‘b’, or adjectives that are often used to describe ocean.
The Magic of COM Libraries
You’ll be using two COM libraries to do all the heavy lifting for you.
The first is the Microsoft XML parser library which also includes the ability to query a website and retrieve data.
The second is the Microsoft Script Control
Retrieving the Web Data
The first thing you’ll need to do is set a reference to the Microsoft XML parser library in the Excel Visual Basic Editor (VBE):
You should find the Microsoft XML library in the list of References.
Its exact location may differ on your computer.
Once you’ve done that, in a new module, enter the following VBA function.
You can see it accepts a web URL as an input parameter and will return some text (as a string).
So, what’s it doing?
- The first couple of lines declare some constants to make the code easier to read and avoid including any “magic numbers”.
- You then declare a variable as the Microsoft XML parsing library and instantiate it.
- Next, you’ll open a connection to the web-service with the URL, request that the data come back in JSON format and send the request.
- Now we wait. We’ll just staying in holding pattern until the request (declared as a Microsoft XML object) tells us it’s finished.
- The function finally returns the JSON it retrieved from the web-service.
Testing Your Code
Let’s test it out.
Enter this routine above the GetResponse() function.
Remember, we’re using the Datamuse web-service to test retrieving JSON data. This query says:
Give me a list of adjectives used to describe the word ‘ocean’.
Let’s test it out.
Enter the following test routine above the GetResponse() function.
Remember, we’re using the Datamuse web-service to test retrieving JSON data.
This query says: Give me a list of adjectives used to describe the word ‘ocean’.
If you pasted this URL into your web browser, it’d look something like this:
Making Sense of JSON
Okay, we’re half-way there.
You’ve got a clean and simple way to retrieve data from a web-service.
Now you just need an efficient way to parse the JSON and load it into Excel.
Enter the Script Control . . .
Now set a reference to the Microsoft Script Control in the Excel VBE:
The script control has several other purposes, including the interpretation and execution of scripts in a host application.
But for your purposes, you just need it to parse JSON.
Enter the following function in your module:
Let’s have a look at what it’s doing . . .
- First you declare your ScriptControl variable and set the Language property to «JScript», Microsoft’s flavour of JavaScript.
- Next you declare another Object variable. You then use ScriptControl’s Eval to parse the JSON into a structure you can interrogate.
- Here we want to populate an array with the Word collection in the items object, but first we use a helper function called GetItemCount to count the number of elements in the items object to size the array correctly.
Yes, you could use ReDim Preserve to keep resizing the array but it’s slow and inefficient.
I’ve listed the GetItemCount helper function below. - Next you loop through each element in the items object and assign each successive value to your array.
- Finally, just for good measure, return the number of elements found.
Now you have a simple routine to parse your JSON data, enter the specific data you want into an array and tell you how many items it found.
What’s in the JSON?
Before we look at the code in more detail, let’s take a quick peek at what’s inside that JSON object we retrieved from the web-service.
Remember, what was returned is a JSON object, not a VBA array.
However, you can peer inside the object as well as loop through it to pluck out data you’re interested in.
When the MS Script Code does is convert the blog of JSON text you saw in the browser screenshot above to something structured.
If you looked at it in the VBE Locals window, it would look like this:
- Notice that the structure “looks” like a regular array with a hierarchy.
- But remember, it’s really a JavaScript object.
Okay, to be pedantic, it’s a JScript – Microsoft’s ‘version’ of JavaScript – object.
Now, back to the code . . .
- The first few lines declare a variable, script, as being a shiny new MS Script Control object and tells it you want to use “JScript”. The other alternative is “VBScript”.
- You then define an object variable called items (original, I know) to hold the data structure. This is then populated by asking the Script Control to evaluate the JSON.
- As this isn’t a true array you can’t use UBound() to determine how many items it has. Instead, you can use a helper function to loop through the Items object and count them.
You’ll then use this number to resize the output array, ready to be populated. This also helps avoid using ReDim Preserve which can be a tad on the slow side. - You then loop through each top-level element in the object, retrieving the value for “word” and passing it to the VBA array.
Just a minute . . .
How did you know the element you wanted was called “word”?
It’s right there in the JSON string that you retrieved earlier.
Remember, JSON is composed of a couple of simple data structures, the main one being an unordered set of name/value pairs (essentially a dictionary).
Now that you’ve looped through the object and populated the array that was passed by reference, the function returns a count of how many items were found.
Now let’s see it in action.
Update the Test_GetResponse routine so it looks like this:
Let’s pull this code apart. . .
- First you declare some variables you need for this process. One to hold the JSON data returned from the webservice (json), an array to hold the values parsed from the JSON object (words) and a range object (Target) so you can paste the results to a worksheet.
- Next you call the GetResponse function, passing the URL and have the resulting JSON assigned to the json variable.
- Finally, you call the ParseJson function, passing the JSON object and the array to be populated.
If this returns a value greater than zero, you swing into action and define where you want the list of words entered and assign the array to that range.
The eagle-eyed among you probably noticed that you’re also flipping the array using the worksheet Transpose function before assigning it to the Target range.
Meanwhile, you should now have a list of words in worksheet one.
You could have used this to populate a listbox, create a report, anything you’d normally do with extract data.
Over to You
Admittedly, the JSON data structure used in this example was not very complicated. However, the same principles apply to more complex data structures.
Take your time to example the returned data in the Locals window and you’ll be in a better position to parse even the most complicated JSON data.
What challenges have you faced importing and processing JSON data into Excel. How did you solve them?
Let me know in the comments below.
If you’re still wrestling with some JSON data let me know about that too.
JSON (Javascript Object Notation) is the most used data exchange format nowadays. Microsoft Excel doesn’t have built-in support for importing JSON to excel or exporting excel data to JSON.
VBA-JSON is an excellent library for parsing JSON in VBA. Lets see how to handle JSON in Excel VBA. If you’re new to JSON then read JSON tutorial for beginners
Prerequisites
- Save your excel file as Macro-Enabled workbook (Refer screen shot below)
- Enable macros if they are not already enabled. You can enable it by cliking on file > options > Trust Center > Trust Center Settings > Enable all macros
Getting Started
- Download VBA JSON latest version from here
- Extract it, open VBA code editor in excel (Alt + F11) and import the library as shown in the gif below.
- Add a reference to Microsoft scripting runtime. (Tools > references > select)
-
Add a reference to MSXML (Tools > references)
Select appropriate version based on your PC :
1. Microsoft XML, v 3.0.
2. Microsoft XML, v 4.0 (if you have installed MSXML 4.0 separately).
3. Microsoft XML, v 5.0 (if you have installed Office 2003 – 2007 which provides MSXML 5.0 for Microsoft Office Applications).
4. Microsoft XML, v 6.0 for latest versions of MS Office.
Import JSON to Excel
This library provides a simple method ParseJson
to parse JSON string into a dictionary object which can be used to extract data. Let’s see an example.
I’m using fake data from http://jsonplaceholder.typicode.com/ which is an API service with fake Json data.
We’ll be pulling user data from http://jsonplaceholder.typicode.com/users by making a GET request which responds with Json data.
Read more about GET requests in VBA here
Next, we’ll parse that Json and import it to excel. Code for importing data looks like this :
Public Sub exceljson() Dim http As Object, JSON As Object, i As Integer Set http = CreateObject("MSXML2.XMLHTTP") http.Open "GET", "http://jsonplaceholder.typicode.com/users", False http.Send Set JSON = ParseJson(http.responseText) i = 2 For Each Item In JSON Sheets(1).Cells(i, 1).Value = Item("id") Sheets(1).Cells(i, 2).Value = Item("name") Sheets(1).Cells(i, 3).Value = Item("username") Sheets(1).Cells(i, 4).Value = Item("email") Sheets(1).Cells(i, 5).Value = Item("address")("city") Sheets(1).Cells(i, 6).Value = Item("phone") Sheets(1).Cells(i, 7).Value = Item("website") Sheets(1).Cells(i, 8).Value = Item("company")("name") i = i + 1 Next MsgBox ("complete") End Sub
Code explanation
- First, define JSON as an object and make a GET request to JSON API
- JSON data received in the response is parsed by passing it into
ParseJson
method. - parsed data is converted into a collection of dictionaries.
- Loop through the collection to get each user’s details and set its values to the first sheet.
Running above code looks like gif below.
Reading JSON from a file
In the same example above, If you want to read JSON data from a local file then you can use FileSystemObject
to read all text in the file and then pass it to ParseJson
method.
Dim FSO As New FileSystemObject Dim JsonTS As TextStream Set JsonTS = FSO.OpenTextFile("example.json", ForReading) JsonText = JsonTS.ReadAll JsonTS.Close Set JSON = ParseJson(JsonText)
Export Excel to Json
VBA-JSON provides another method ConvertToJson
which can be used to convert excel data into JSON. Here’s an example.
Sample data with Name, Phone and Email is present in second sheet. Let’s convert it into JSON
Code for this looks like :
Public Sub exceltojson() Dim rng As Range, items As New Collection, myitem As New Dictionary, i As Integer, cell As Variant set rng = Range("A2:A3") 'Set rng = Range(Sheets(2).Range("A2"), Sheets(2).Range("A2").End(xlDown)) use this for dynamic range i = 0 For Each cell In rng Debug.Print (cell.Value) myitem("name") = cell.Value myitem("email") = cell.Offset(0, 1).Value myitem("phone") = cell.Offset(0, 2).Value items.Add myitem Set myitem = Nothing i = i + 1 Next Sheets(2).Range("A4").Value = ConvertToJson(items, Whitespace:=2) End Sub
Code Explanation
- First, define
rng
as range and set it to data range. ConvertToJson
method takes a dictionary collection or array as parameter. So we should pass our data as a collection.- A Dictionary is an object with keys and values just like JSON but doesn’t support multiple items like arrays or collections, so we create a dictionary for each item and push it into an array or a collection.
- Define a dictionary and a collection, loop through the range and set each row’s data into
myitem
- Push
myitem
into collection and set it to nothing, because we are using the same dictionary to add next row’s data and push it to collection again.
Finally pass items
collection to ConvertToJson
method which returns a JSON string.
Running above code looks like gif below
Export Excel to JSON file
In the same example above, If you want to export excel data to JSON file then It can be done by opening a file for output by specifying the path of the file and printing data in it. Sample code below, Running this would save a JSON file in the current workbook’s folder.
Public Sub exceltojsonfile() Dim rng As Range, items As New Collection, myitem As New Dictionary, i As Integer, cell As Variant, myfile As String Set rng = Range("A2:A3") 'Set rng = Range(Sheets(2).Range("A2"), Sheets(2).Range("A2").End(xlDown)) use this for dynamic range i = 0 For Each cell In rng Debug.Print (cell.Value) myitem("name") = cell.Value myitem("email") = cell.Offset(0, 1).Value myitem("phone") = cell.Offset(0, 2).Value items.Add myitem Set myitem = Nothing i = i + 1 Next myfile = Application.ActiveWorkbook.Path &amp; "data.json" Open myfile For Output As #1 Print #1, ConvertToJson(items, Whitespace:=2) Close #1 End Sub
Export Excel to Nested JSON
Above code can be modified a bit to get a nested JSON as output. Just add dictionary in another dictionary so that it creates a nested JSON. code looks like this :
Public Sub exceltonestedjson() Dim rng As Range, items As New Collection, myitem As New Dictionary, subitem As New Dictionary, i As Integer, cell As Variant Set rng = Range("A2:A3") 'Set rng = Range(Sheets(2).Range("A2"), Sheets(2).Range("A2").End(xlDown)) use this for dynamic range i = 0 For Each cell In rng Debug.Print (cell.Value) myitem("name") = cell.Value myitem("email") = cell.Offset(0, 1).Value myitem("phone") = cell.Offset(0, 2).Value subitem("country") = cell.Offset(0, 3).Value myitem.Add "location", subitem items.Add myitem Set myitem = Nothing Set subitem = Nothing i = i + 1 Next Sheets(2).Range("A4").Value = ConvertToJson(items, Whitespace:=2) End Sub
Running above code looks like image below
Using array of strings and objects in JSON
One of our readers asked me how to use an array of strings and objects inside the JSON.
Here’s how to do it:
Public Sub exceltojson() Dim rng As Range, items As New Collection, myitem As New Dictionary, i As Integer, cell As Variant, objectContainer As New Dictionary, arrayContainer As New Collection Dim j As Integer 'Set rng = Range("A2:A3") Set rng = Range(Sheets(1).Range("A2"), Sheets(1).Range("A2").End(xlDown)) 'use this for dynamic range i = 0 For Each cell In rng myitem("id") = cell.Value myitem("name") = cell.Offset(0, 1).Value 'tags tagsString = cell.Offset(0, 2).Value tagsCollection = getCollectionFromString(tagsString) myitem.Add "tags", tagsCollection 'ingredients ingredientsString = cell.Offset(0, 3).Value weightsString = cell.Offset(0, 4).Value ingredientsUnit = cell.Offset(0, 5).Value ingredientsCollection = getCollectionFromString(ingredientsString) weightsCollection = getCollectionFromString(weightsString) j = 0 For Each ingredient In ingredientsCollection objectContainer("ingredientnaam") = ingredient objectContainer("eenheid") = ingredientsUnit objectContainer("hoeveelheid") = weightsCollection(j) arrayContainer.Add objectContainer Set objectContainer = Nothing j = j + 1 Next myitem.Add "ingredienten", arrayContainer 'Reset values Set arrayContainer = Nothing j = 0 items.Add myitem Set myitem = Nothing i = i + 1 Next Sheets(1).Range("A6").Value = ConvertToJson(items, Whitespace:=2) End Sub Function getCollectionFromString(val) getCollectionFromString = Split(val, ", ") End Function
Running above code looks like image below
Wrapping up
Read official documentation of VBA-JSON here and use VBA-Dictionary for Mac Support.
Related articles :
- Complete JSON tutorial here – JSON for beginners
- Handling CSV in VBA
If you have any questions or feedback, comment below and please use CodingisLove Bin for sharing your code.
- Author
- Recent Posts
A CA- by education, self taught coder by passion, loves to explore new technologies and believes in learn by doing.
In this article, we will learn how to perform the How can you extract a nested JSON value in VBA Excel using VBA-Json Parse Nested Json package.
In this article, I will explain to you a step-by-step for performing Parsing complex JSON Data using VBA JSON in Excel VBA.
You can download the source code at the bottom of the post
Recently I’m working on a Excel VBA project in which I need to call the rest and parse the json response. I google this question and but can’t find any good article which covers this topic with complex json response. so that I decided to write an article on Parsing Nested Arrays using VBA and JSON.
VBA Excel doesn’t have a built JSON parser for us , that’s why are going to use VBA-tools JSON parser package to parse the JSON that we receive after making a rest api calll.
Go to the GitHub link : https://github.com/VBA-tools/VBA-JSON
and download code.
And open your Excel sheet and go to the developer tool and visual basic and then import the JSON converter that we download from Github.
Go to File and click on Import file and naviate to the folder that and select the JsonConverter.bas And click on Open.
So now go back to the visual basic IDE and click on Tools and select references and Microsoft scripting runtime references in our project.
So let write code for parsing json.
Simple Json Parsing Example
Private Sub ParseSimpleJson() Dim JsonObject As Object Dim strResponse As String strResponse = "{""name"": ""johny"", ""address"": { ""country"": ""USA"",""city"": ""New York City"" } }" Set JsonObject = JsonConverter.ParseJson(strResponse) MsgBox ("User name :" & JsonObject("name") & " User Country: " & JsonObject("address")("country")) End Sub
1. Parsing Nested Arrays using VBA and JSON
Sample json
{ "apidata": { "success": true, "data": { "music_events": { "Magic": { "users": ["Tayler", "Ally"], "event_status": "Pending", "event_sites": { "ticketbet": { "odds": { "h2h": ["1.86", "1.99"] }, "last_update": 1488956952 }, "stream411": { "odds": { "h2h": ["1.70", "2.10"] }, "last_update": 1488957101 }, "darkmusic": { "odds": { "h2h": ["1.83", "1.98"] }, "last_update": 1488957104 }, "lastride": { "odds": { "h2h": ["1.83", "2.00"] }, "last_update": 1488957115 } } } } } } }
Our goal is to get the music_events, users details into table and the event_sites data into a separate table.
VBA Code for parsing above json
Private Sub ParseNestedJson() Dim JsonObject As Object Dim strResponse As String Dim music_events, k, users, v, event_sites strResponse = Sheet1.Range("A1").Value Set JsonObject = JsonConverter.ParseJson(strResponse) Set music_events = JsonObject("apidata")("data")("music_events") For Each k In music_events Debug.Print "event", k Set users = music_events(k)("users") For Each v In users Debug.Print , "participant", v Next v Set event_sites = music_events(k)("event_sites") For Each v In event_sites Debug.Print , "site", v Next v Next 'MsgBox ("User name :" & JsonObject("name") & " User Country: " & JsonObject("address")("country")) End Sub
2. Parsing complex JSON Data using VBA JSON
Sample Json
{ "UniqueId": "{344DSD-343-34D-343-23SDSDSD}", "from": "2021-01-16", "to": "2021-01-22", "data": [ { "date": "2021-01-16", "person": "{34343DS-343-3434-343-SFDSS343}", "personName": "Rohit Smith", "company": "{SDSD344-343-343-343-3FDFDFD}", "companyName": "Appsloveworld pvt ltd", "minutes": "400", "task": [ { "name": "Training", "code": "TRN", "minutes": "120" }, { "name": "Human Resources", "code": "HR", "minutes": "150" }, { "name": "Yoga", "code": "YG", "minutes": "15" }, { "name": "Lunch", "code": "", "minutes": "30" } ] } ] }
There may be any number of ‘data’ records, as well as any number of ‘tasks’ within each “data” including zero.
we want a row in the spreadsheet for each activity, with the task name and other data outputted next to that day’s task.
VBA Code for that
Sub NestedJsonExample() Dim ts, act Dim Json As Object, c As Range, strResponse As String 'reading json from a worksheet cell... strResponse = Sheet1.Range("B1").Value Set Json = JsonConverter.ParseJson(strResponse) Set c = ActiveSheet.Range("C5") 'loop over timesheets For Each ts In Json("data") 'loop over timesheet activities For Each act In ts("task") c.Resize(1, 11).Value = Array(Json("UniqueId"), Json("from"), Json("to"), _ ts("date"), ts("personName"), ts("companyName"), _ ts("minutes"), act("name"), act("code"), _ act("minutes")) Set c = c.Offset(1, 0) Next act Next ts End Sub
Download Source Code
Some information Regarding VBA
Excel VBA is the programming language of Microsoft Excel like for Microsoft Office projects like Word and PowerPoint.
VBA is the truncation for Visual Basic for Applications. It is an occasion driven programming language from Microsoft. Which is currently fundamentally utilized with Microsoft Office applications like MS-Excel, MS-Word and MS-Access. It helps in the making of tweaked applications and their answers, which improve the abilities of those applications. The benefit of this component is that we don’t have to introduce Visual Basic on our PC yet introducing Office assists us with accomplishing our target.
We can utilize VBA in all renditions of Office from MS Office 97 to MS Office 2013. You can likewise explore different avenues regarding other present day forms of Office that are accessible with them. Dominate VBA is the most famous of all VBA and the benefit of utilizing VBA is that we can assemble an amazing asset utilizing Linear Programming.
Visual Basic is a programming language that accompanies a coordinated advancement climate. Planned by Microsoft, the Visual Basic download makes coding a basic and pleasant experience. Reasonable for all clients, including fledglings and specialists, this language is object-driven and gives you admittance to sentence structure developments and an information base of components. You can fabricate a scope of Windows applications and front-end frameworks.
What is Windows Visual Basic?
Visual Basic is an article driven improvement climate and PC programming language made by Microsoft. The framework gives a graphical UI that permits them to alter the code by relocating components, permitting clients to change the appearance and conduct of the application. The article arranged language depends on BASIC and is considered appropriate for amateurs to code.
Microsoft expected to improve on the language and backing quicker coding. That is the reason it is known as RAD or Rapid Application Development System. With its assistance, coders can model applications prior to thinking of them in a more effective however troublesome dialect. What’s more, Virtual Basic likewise gives punctuation that is more clear and data set associated.
The post Simple way to Parse JSON with Excel VBA appeared first on Software Development | Programming Tutorials.
Read More Articles
- Excel VBA macro using iTunes search API — fastest way to query & parse JSON results
- How to parse JSON with VBA without external libraries?
- JSON VBA Parse to Excel
- Parse JSON with VBA (Access 2010)
- Quickest Way to open an excel file with VBA
- Excel VBA throws overflow error with a simple division
- Simple way to refresh power pivot from VBA in Excel 2010?
- Excel VBA Arrays: Is there a simple way to delete a dataset by the index?
- Parse simple two dimensional JSON array in VBA without external libraries
- MS Excel 2003 — Simple unselect question in Excel VBA when dealing with shapes
- Best way to distribute Excel spreadsheet with VBA
- Simple recursive function in VBA with Excel not returning expected result
- Fastest way of Parsing Json to Excel using VBA
- Is there a simple way to parse comma separated Key:Value pairs in Excel, Power Query or VBA if the values contain unescaped commas?
- parse a string with VBA on EXCEL
- Parse local webpages with Selenium in VBA Excel fails
- How can I parse json in Excel vba without using microsoft scripting runtime?
- How to fix and extract google CSE JSON API results with excel vba
- Using VBA in Excel to retrieve a Json from an API (problem with the API password)
- VBA Json Parse response with JsonConverter
- Parsing Google Books JSON to obtain book info by entering ISBN in EXCEL with VBA
- Review my simple VBA script with built-in Excel function
- Excel VBA out of memory error with simple line of code
- get data from a JSON string with VBA Excel
- Best way to get column totals on multiple columns with varying row sizes using MS Excel VBA
- Faster Way to copy files from one folder to another with Excel VBA
- VBA code in excel operates inconsistently with very simple code
- VBA Excel — Problems with a simple macro to auto-fill cells for a budgeting spreadsheet I’m attempting to make
- excel to json with vba
- With Excel VBA is there a way to copy an entire worksheet’s NumberFormat to a new worksheet?
- Stop IE from loading further when required data is there
- VBA Excel : Populate TextBox with specific string of text when Option Button is selected
- Object Defined Error in Data from Internet Pull
- Range.Find does not find exactly what I’m searching for
- How to disable arrow keys in ListView control in VBA with Excel
- Run-time error ‘1004’: The PivotTable field name is not valid
- Weird activecell.offset output
- Move files from multiple folders to a single folder
- moving range of values from one sheet to another
- VBS code not working in HTA (WScript declaration)
- VBA Form Is (automatically) Tied/Bound To ActiveWorksheet When calling Form.Show vbmodeless
- Make the second combo-box item selected based on item selection of first combo box
- How to check the same values between 2 table (MSAccess)
- How do you run a Word mail merge macro from Excel?
- Stop procedure if condition is not met using a global variable in VBA
- Setting default values in Data Validation drop-down list
- Pivot table expand and substract or Show/Hide Details using Excel VBA
- Using one or two dimensional arrays in Excel VBA
- Summing up a column based on multiple Criterion
- Calculate Time of each row excel