Excel Web Query — What in the world is that? If you are like the other 99.9% of MS Excel users, you probably have never heard of microsoft excel web queries (note: statistic made up).
Excel web queries are powerful! Web queries are basically like having a web browser built into Excel that attempts to format the content, putting individual pieces of data into separate cells. You can then use Excel formulas (like =A1/B2) to work directly with the data you’ve downloaded. And you don’t have to know anything about perl, cgi, php, javascript, etc.
Getting Started with Sample Excel Web Queries
The example web queries that come with Excel are usually for importing stock quote data into Excel. For more information about using web queries to download Stock Quotes into Excel, take a look at the Excel Stock Quotes Template.
Although importing stock quotes into Excel happened to be the reason I came across Excel web queries, I have since found many other ways to use them.
It is not the web query itself that is the secret that this article is about. Instead, it’s the ability to make a DYNAMIC Web Query!
By dynamic, I’m not talking about using «refresh» to update the data. What I mean is the ability to change the parameters of the web query, such as typing in a new stock symbol into a cell and having the table update automatically.
If you find yourself using the internet to gather data by filling out forms and copying and pasting data over and over, an excel web query might be the solution to your monotonous woes. It’s not always going to be better, and it’s not always going to work, but it’s worth a try, especially since a simple web query could boost your productivity!
Excel Web Query «.iqy» Files
The real key to creating a dynamic excel web query is to create your own «.iqy» file. In it’s basic form, the «.iqy» file is simply a TEXT file with three main lines:
WEB 1 https://www.thedomain.com/script.pl?paramname=value¶m2=value2
You can create the file using a simple text editor! Most text editors will save a new file with the .txt extension, but you can change the extension to .iqy after you create your text file. If it is already a .iqy file and you want to edit it, you may need to right-click on it to select NotePad or WordPad (or some other text editor) to open it.
The third line is the important one! It’s simply the URL or web page address. Notice that it contains two parameters. If you don’t know what parameters are for, just browse the web for a while and pay attention to what shows up in the address bar of your browser. Go to Google.com for instance, and look up «excel web query».
Parameter name / value pairs are listed after the «?» in the URL and are separated by an «&«.
Make the Web Query Dynamic
To make the query dynamic, replace the value of each parameter in the web query file (queryname.iqy) with:
["paramname","Enter the value for paramname:"]
Want to see how this would apply to a Google search? The form that I used above consists of HTML code that looks like this:
<form action="https://www.google.com/search">
<input type="text" name="q" value="excel web query">
<input type="submit" value="Search Google">
</form>
Notice that «q» is the name of the parameter, and the action tells you what the URL should be. The dynamic web query file for a simple google search would look like this:
WEB 1 https://www.google.com/search?q=["keyword","Enter the Search Term:"]
Let’s Create a Web Query
- Open up a text editor and copy the Google example. Save the file as GoogleSearch.iqy
- Open up Excel and enter a search term in Cell B3
- Open the web query file you just saved (GoogleSearch.iqy) by double-clicking on the file, or open Excel and go to Data > Existing Connections > Browse for More.
- When prompted for the search term, enter =B3. If that doesn’t work, just enter a search term for now.
Play around with the web query options and properties by right-clicking within the query results and selecting Parameters or Data Range Properties or Edit Query.
When you go to Edit Query, you can choose to select a specific portion or table within the page or the entire page (by clicking on one of the yellow arrow boxes in the preview). Then, you can save your new query as a .iqy file and look at what parameters it uses.
WEB 1 https://www.google.com/search?q=["q","Enter the Search Term:"] Selection=EntirePage Formatting=RTF PreFormattedTextToColumns=True ConsecutiveDelimitersAsOne=True SingleBlockTextImport=False DisableDateRecognition=False DisableRedirections=True
When you have the query looking just the way you want it, save the Excel web query (an icon or button for saving the web query is in the «Edit Query» window. When you save the new query, your «.iqy» file will include the options you have selected.
Other Example Web Queries
Below are a few examples of .iqy files that you might try. Websites often change, or they may change the way their APIs work, so these examples might not work forever.
Example 1: Get historical daily stock prices from Investopedia. Note that the start and end dates are text, so if you are setting up date parameters in Excel, the cells you link to need to be text rather than date values. This query uses 3 parameters: Symbol, StartDate, EndDate.
See this article for other sources for historical stock quotes.
Example 2: Get data from a published Google Spreadsheet. See my article Excel Stock Quotes in Excel to see how you can publish a Google Spreadsheet and then use a web query to get the data into Excel.
Become a Web Query Wiz
Take a few minutes to think about what mundane or repetitive tasks YOU do on the internet, particularly cases where you go to the same sites over and over to get data. If you find yourself copying and pasting or re-typing information into Excel so that you can perform calculations with the data, you may have found an ideal use for an excel web query.
There are two main things that you need in order to become a Web Query Wiz.
- A working knowledge of Excel formulas. By this, I mean the ability to use text-manipulation and other formulas. A GREAT book for both learning and reference (and the one that I use all the time) is John Walkenbach’s «Excel Formulas». If you are an avid Excel user, it will probably be the best investment you will make for a long time! The only problem I have with the book is that I need TWO copies — one for work and one for home.
- A working knowledge of HTML. This is the hard part unless you are a web designer or programmer. The articles listed in the side bar will help to some extent, but ultimately this issue was the reason why I didn’t write a more detailed tutorial. Something that will be a great help is to learn about how <form> tags work. Here is my favorite reference.
If you found this article interesting, be sure to check out some of my other Excel tips.
— Jon Wittwer, President
Vertex42, LLC
Московская биржа предоставляет возможность скачивать котировки и другую информацию по ценным бумагам при помощи специальных ссылок.
Если мы введем такую ссылку в Power Query, создав запрос из интернета, то получим интересующие нас котировки прямо в Excel.
Web queries offer a handy way to import data from selected tables into a worksheet—but Excel isn’t as accommodating as it should be, especially when it comes to parameters. These tricks will let you work around Excel’s limitations so that you can get the data you need.
An Excel Web query allows you to bring data from a Web site
into an Excel worksheet. It will find any tables on the Web page and let you
select the ones containing data you want to put into your worksheet, allowing
for dynamic updates from the Web page. Web queries are not just useful for
pulling information from standard HTML pages. They can also be used quite
nicely in situations where a standard ODBC connection would be difficult or
impossible to create or maintain, such as a worksheet that’s used by
salespeople around the country.
We’re going to work through a basic example to get a feel
for how Web queries operate. Then, we’ll look at a couple of tricks that allow
you to work around some Excel limitations and see how a little VBA code can
give you the results you want. You can download this Excel workbook to see a
sample implementation of these techniques.
Getting started
To demonstrate the process, we’re going to start with a
simple Web query using Yahoo! Finance historical stock prices. This is a great
example because the data we’re interested in is presented in a plain, tabular
format. Another advantage is that the URL contains the stock symbol (GOOG, in
this case), so it will be easy to manipulate via a VBA macro, and it has little
confusing information in it. Finally, this Web query doesn’t put important
information in images or through links.
To create the Web query:
- Select
the first cell in which you want results to appear. - Choose
Data | Import External Data | New Web Query to open the dialog box shown in Figure A.
|
Figure A |
 |
- Enter
the URL to query in the Address area and click the Go button (Figure B).
|
Figure B |
 |
- Select
the table you want to use for the query (Figure
C).
|
Figure C |
 |
- Click
the Import button.
That’s it. The data is now in your worksheet (Figure D).
|
Figure D |
 |
Customizing the query
After you create a Web query, you can customize it to meet
your needs. To access Web query properties, right-click on a cell in the query
results and choose Edit Query. (You can also click Edit Query on the External
Data toolbar or choose Data | Import External Data | Edit Query.) When the Web
page you’re querying appears, click the Options button in the upper-right
corner of the window to open the dialog box shown in Figure E. The options here allow you change how the query interacts
with the Web page itself.
|
Figure E |
 |
In addition, you have the same choice of Data Range
options that you have with other external data queries, such as ODBC queries. Just
right-click on a cell in your query results and choose Data Range Properties
(or click Edit Query on the External Data toolbar or choose Data | Import
External Data | Data Range Properties) to open the dialog box shown in Figure F. You’ll probably want to
change the Data Range’s name from the default to a name you
can easily access through a macro. In this example, we’ll rename the Data Range
to Stock Prices.
|
Figure F |
 |
Parameters
Unfortunately, working with parameters is not as straightforward
as it could be. The wizard that created the sample query above doesn’t allow
you to put in parameters. When it fetches the Web page, it will escape the parameter
identifiers in the URL and return a page without the right results. For many
Web pages, this prevents you from selecting the table you want to use for the
import, so you’ll need to work around this Excel limitation. The Web query
parameters let you use set values, get the values from a worksheet range,
prompt the user for input, or set values programmatically via VBA macros.
One workaround is to create your query as outlined above and
then run a VBA macro to dynamically change the Connection property of the query
to provide the correct URL for Web pages that use the GET data. In this
example, we could use code similar to
Listing A
.
For Web pages that use GET
data, this is a perfectly fine solution. For Web pages that use POST data, this
is not a solution. Those scenarios will require a little bit of manual labor. You’ll
need to open a text editor and create an IQY (Internet Query) file as a plain
text file to use as the basis for your data import. The IQY file should have four
lines:
Type of Query
Query Version
URL
POST Parameters
The Query Type line should just be WEB and the Query Version
line can be whatever you want (1 is just fine). The URL line should be the URL
itself. The parameters should be in the following format:
Parameter1=Value1&Parameter2=[“Value 2”, >”Please input a value for Parameter 2: “]
This is where you can work a couple of tricks. With that
sample parameter line, “Value1” will be passed as the value for Parameter1, but
the user will be prompted with the phrase “Please input a value for Parameter
2:” to provide the value for Parameter2. This works with GET as well as POST
queries. The prompt text is optional.
In this example, we just need a single parameter–the stock
symbol. So our Web Query file looks like this:
WEB
1
http://finance.yahoo.com/q/hp
s=[“Stock Symbol”]
Once you’ve created the query file, you can use it in the
worksheet. Choose Data | Import External Data | Import Data and point the Open dialog
box to the query file you created. You’ll be prompted to specify the stock
symbol to be used. Your query results will then show up in the worksheet.
One final trick is to use this query file to fool the import
wizard into allowing you to select the exact table of desired data and still
use parameters. Once the query file has been loaded, you can edit the query in
Excel. The query won’t work because the editor won’t pass our values along
properly. It will, however, show you the Web page, and from there you can
navigate or search or whatever you need to do to see the page with the table so
you can select it. Once you’ve selected the table (just like creating a Web
query through the wizard), you can edit the parameters to use the desired methodology
(prompts, hard-coded values, or data from a worksheet range). You can also now
set these values through VBA code.
Once the Web Query has been created from the file, you no
longer need the file. It doesn’t have to be distributed with the Excel worksheet.
Working with the query through
VBA
The DataRange gets added to the QueryTables collection, which is a
member of the Worksheet object, and can be referenced by index
number or by the DataRange name. In our
example, ThisWorkbook.Sheets(“Web Query”).QueryTables(“Stock
Prices”) refers to our Web query. The most common use of VBA with
the Web Query is to have its Refresh()
method force the data to be refreshed from the source and reloaded into the worksheet.
You can also use the SetParam() method on the Parameter property of the QueryTable object to manually set (or
prompt the user to input) the value of the parameter (the Value property is read only). The sample piece of code in
Listing B
takes the stock symbol as a string, puts it into the
parameter, and then updates the query’s data.
Если требуется регулярно обращаться к обновляемым данным с веб-страницы, создайте веб-запрос. На рис. 1 представлен сайт https://news.yandex.ru, показывающий курс доллара. [1]
Рис. 1. Использование диалогового окна Создание веб-запроса для указания данных, которые следует импортировать; чтобы увеличить изображение кликните на нем правой кнопкой мыши и выберите Открыть картинку в новой вкладке
Скачать заметку в формате Word или pdf, примеры в формате Excel
Термин «веб-запрос» немного неточен, так как с его помощью можно получать данные не только из Сети. Можно также выполнять веб-запрос к HTML-файлу, расположенному на локальном компьютере, или к файлу, хранящемуся на сетевом сервере либо на веб-сервере в Интернете. Чтобы получить информацию с веб-сервера, необходимо подключиться к Интернету. Когда данные получены, соединение с Интернетом для работы с ними больше не нужно (конечно, если вы не собираетесь обновлять эти данные).
Создать веб-запрос, обеспечивающий получение информации, которую позже можно будет обновить одним щелчком мыши, можно следующим образом.
- Выполните команду Данные –> Получение внешних данных –> Из Интернета, чтобы открыть диалоговое окно Создание веб-запроса.
- В поле Адрес введите URL сайта и нажмите Пуск. В нашем примере (рис. 1) изображена веб-страница, расположенная по адресу https://news.yandex.ru/quotes/1.html. Обратите внимание: в диалоговом окне Создание веб-запроса открывается браузер (Internet Explorer). В нем вы можете щелкать на ссылках и переходить по сайту, пока не найдете интересующие вас данные. Когда веб-страница отображается в окне Создание веб-запроса, вы видите один или несколько желтых квадратов со стрелками, соответствующих таблицам, находящимся на странице. Еще один такой же квадрат будет соответствовать странице целиком.
- Если щелкнуть на желтом квадрате со стрелкой, он превращается в зеленый флажок, указывающий, что данные этой таблицы будут импортированы. К сожалению, таблицу с курсами валют нельзя выделить, поэтому единственный выход — импортировать страницу целиком.
- Нажмите Параметры, чтобы выбрать формат импорта данных и иные настройки.
- Нажмите кнопку Импорт, чтобы отобразить диалоговое окно Импорт данных. Укажите местоположение для импортированной информации. Это может быть, как ячейка на актуальном листе, так и новый лист. Нажмите Свойства, если хотите изменить настройки импорта по умолчанию.
- Нажмите Ok, и Excel импортирует данные.
Результат импорта (рис. 2) не совсем то, чего мы хотели. Нас интересовала только таблица валют, а веб-запрос вернул страницу целиком. По умолчанию импортированные данные являются веб-запросом. Чтобы обновить информацию, щелкните правой кнопкой мыши на любой ячейке в импортированном диапазоне и в контекстном меню выполните команду Обновить.
Рис. 2. Информация, полученная с помощью веб-запроса
Если вы не собираетесь создавать обновляемый запрос, укажите это в шаге 5 приведенного выше порядка действий. В диалоговом окне Импорт данных нажмите кнопку Свойства и снимите флажок Сохранить определение запроса.
Встроенная в Excel возможность работы с веб-запросами позволяет находить в HTML-документе таблицы (обозначаемые HTML-тегом <TABLE>). Однако веб-дизайнеры для отображения табличной информации все чаще пользуются каскадными таблицами стилей (CSS). Как понятно из данного примера, Excel не распознает такие таблицы и не показывает желтого квадрата со стрелкой, поэтому получить таблицу вы не сможете — вам может потребоваться получить целый документ, а потом удалить (или скрыть) все данные, кроме интересующей вас таблицы.
Еще один способ отобразить данные с веб-страницы на рабочем листе напрямую открыть URL в Excel, воспользовавшись командой Файл –> Открыть. Просто введите URL полностью в поле Имя файла и нажмите Открыть (рис. 2). Результаты могут различаться в зависимости от макета веб-страницы. Как правило, итог удовлетворительный. Иногда вы получаете немного лишней информации. Обратите внимание: такие данные не обновляются. Если информация на веб-странице изменится, нужно будет закрыть книгу и снова воспользоваться командой Файл –> Открыть. Мне не удалось этого сделать. Excel сначала писал, что открывает страницу в режиме защищенного просмотра, долго работал, а потом выдал, что книга повреждена, и открыть ее нельзя((
[1] По материалам книги Джон Уокенбах. Excel 2013. Трюки и советы. – СПб.: Питер, 2014. – С. 194–196.
Excel-Web-Query
A library for web queries as used by Microsoft Excel.
What is a web query and what is this library for?
A web query is a file that tells Microsoft Excel where and how to fetch data from the web.
This library lets you create web queries in the appropriate format. This allows your applications
to wrap URLs in a way that Microsoft Excel understands. Microsoft Excel can then import data by querying
these URLs.
How to use web queries in Microsoft Excel
- Create a web query and save it with the file extension «iqy»
- Open Microsoft Excel, go to the «Data» menu, select «Get External Data» and
«Run Web Query» / «Run Saved Query», select your web query file and import it - Microsoft Excel will now automatically import the response from the URL in your web
query whenever it imports external data
How to use this library
Add Jitpack as a repository to your project and add a dependency for excel-web-query in order to
use this library (the code snippet below uses Gradle’s Kotlin DSL):
repositories {
maven { url = uri("https://jitpack.io") }
}
dependencies {
implementation("com.github.spartusch:excel-web-query:latest.release")
}
The actual usage of the library is straightforward:
Create a new WebQuery by calling WebQueryFactory.create.
This library is written in Kotlin but tested with Java and Kotlin.
Example on how to create a ResponseEntity for Spring MVC
Java
final var webQuery = WebQueryFactory.create(uri);
final var headers = new HttpHeaders();
headers.set(HttpHeaders.CONTENT_TYPE, webQuery.getContentType());
headers.set(HttpHeaders.CONTENT_LENGTH, String.valueOf(webQuery.getContentLength()));
headers.set(HttpHeaders.CONTENT_DISPOSITION, WebQuery.getContentDisposition("webquery.iqy"));
final var response = new ResponseEntity<>(new ByteArrayResource(webQuery.getContentBytes()), headers, HttpStatus.OK);
Kotlin
val webQuery = WebQueryFactory.create(uri)
val headers = HttpHeaders()
headers[HttpHeaders.CONTENT_TYPE] = webQuery.contentType
headers[HttpHeaders.CONTENT_LENGTH] = webQuery.contentLength.toString()
headers[HttpHeaders.CONTENT_DISPOSITION] = WebQuery.getContentDisposition("webquery.iqy")
val response = ResponseEntity(ByteArrayResource(webQuery.contentBytes), headers, HttpStatus.OK)
How to build and publish this library
Build this library with Gradle and publish it, e.g. locally:
./gradlew publishToMavenLocal
You can also build a base image for Docker: