Key word in context

From Wikipedia, the free encyclopedia

Key Word In Context (KWIC) is the most common format for concordance lines. The term KWIC was first coined by Hans Peter Luhn.^[1] The system was based on a concept called keyword in titles which was first proposed for Manchester libraries in 1864 by Andrea Crestadoro.^[2]

A KWIC index is formed by sorting and aligning the words within an article title to allow each word (except the stop words) in titles to be searchable alphabetically in the index.^[3] It was a useful indexing method for technical manuals before computerized full text search became common.

For example, a search query including all of the words in an example definition («KWIC is an acronym for Key Word In Context, the most common format for concordance lines») and the Wikipedia slogan in English («the free encyclopedia»), searched against a Wikipedia page, might yield a KWIC index as follows. A KWIC index usually uses a wide layout to allow the display of maximum ‘in context’ information (not shown in the following example).

A KWIC index is a special case of a permuted index.^[4] This term refers to the fact that it indexes all cyclic permutations of the headings. Books composed of many short sections with their own descriptive headings, most notably collections of manual pages, often ended with a permuted index section, allowing the reader to easily find a section by any word from its heading. This practice, also known as Key Word Out of Context (KWOC), is no longer common.

In academia, The KWIC analysis result reflects the cognitive–emotional image perceived by the commenters in YouTube. The research studied number of times that the contextual words appeared on the right- and left-hand sides of the French keywords, and the score, which indicates the correlation between the French keywords and contextual French words in Architectural Heritage YouTube Videos.^[5]

• 

  Keyword alongside context (KWAC)

• 

  Keyword in context (KWIC)

• 

  Keyword out of context (KWOC)

References in literature[edit]

Note: The first reference does not show the KWIC index unless you pay to view the paper. The second reference does not even list the paper at all.

• David L. Parnas uses a KWIC Index as an example on how to perform modular design in his paper On the Criteria To Be Used in Decomposing Systems into Modules, available as an ACM Classic Paper
• Christopher D. Manning and Hinrich Schütze describe a KWIC index and computer concordancing in section 1.4.5 of their book Foundations of Statistical Natural Language Processing. Cambridge, Mass: MIT Press, 1999. ISBN 9780262133609. They cite an article from H.P. Luhn from 1960, «Key word-in-context index for technical literature (kwic index)».
• According to Rev. Gerard O’Connor’s Concordantia et Indices Missalium Romanorum, «Most of the concordances produced in recent times and with the aid of computer software use both the KWIC (keyword in context) and KWICn (keyword in center) formats, which lists the keyword, usually highlighted in bold text in a consistent position, within a limited amount of context text, i.e. three [or] four words of the text prior to the keyword and the same amount of text following. This format is extremely useful in that the keyword is easily identified together with its context. … The Concordance of the Roman Missal is produced in both the KWIC and KWICn formats and is noteworthy in that each word form is listed as it appears in the text, that is, it is un-lemmatized.»

See also[edit]

• ptx, a Unix command-line utility producing a permuted index
• Concordancer
• Concordance (publishing)
• Burrows–Wheeler transform
• Hans Peter Luhn
• Suffix tree

References[edit]

KWIC — это аббревиатура от Key Word In Context, наиболее распространенного формата для co. ncordance строк. Термин KWIC впервые был придуман Гансом Петером Луном. Система была основана на концепции, называемой ключевым словом в заголовках, которая была впервые предложена для библиотек Манчестера в 1864 году Андреа Крестадоро.

A Индекс KWIC формируется путем сортировки и выравнивания слов в заголовке статьи, чтобы каждое слово (кроме стоп-слов ) в заголовках для поиска в алфавитном порядке в указателе. Это был полезный метод индексации технических руководств до того, как компьютеризированный полнотекстовый поиск стал обычным.

Например, поисковый запрос, включающий все слова в заголовке этой статьи («KWIC является аббревиатурой от Key Word In Context, наиболее распространенного формата для строк соответствия ») и лозунга Википедии на английском языке (« бесплатная энциклопедия »), поиск которого выполняется на этой самой веб-странице, может дать индекс KWIC следующим образом. Индекс KWIC обычно использует широкий макет, чтобы обеспечить отображение максимальной информации «в контексте» (не показано в следующем примере).

Индекс KWIC — это частный случай переставленного индекса. Этот термин относится к тому факту, что он индексирует все циклические перестановки заголовков. Книги, состоящие из множества коротких разделов с собственными описательными заголовками, в первую очередь сборники справочных страниц, часто заканчиваются разделом переставленного указателя, что позволяет читателю легко найти раздел по любому слову из его заголовка. Эта практика, также известная как KWOC («Ключевое слово вне контекста »), больше не распространена.

• 

  Ключевое слово в контексте (KWAC)

• 

  Ключевое слово в контексте (KWIC)

• 

  Ключевое слово вне контекста (KWOC)

Ссылки в литературе

Примечание: первая ссылка не показывает KWIC index, если вы не платите за просмотр статьи. Вторая ссылка даже не перечисляет статью.

• Дэвид Л. Парнас использует Индекс KWIC в качестве примера того, как выполнять модульное проектирование в своей статье О критериях, которые должны использоваться при разложении систем на модули, доступной как ACM Classic Paper
• Кристофер Д. Маннинг и Хинрих Шютце описывают индекс KWIC и компьютерное согласование в разделе 1.4.5 своей книги «Основы статистической обработки естественного языка». Кембридж, Массачусетс: MIT Press, 1999. ISBN 9780262133609 . Они цитируют статью H.P. Луна из 1960, «Контекстный указатель ключевых слов для технической литературы (kwic index)».
• Согласно Concordantia et Indices Missalium Romanorum преподобного Джерарда О’Коннора, «Большая часть согласования, созданные в последнее время и с помощью компьютерного программного обеспечения, используют форматы KWIC (ключевое слово в контексте) и KWICn (ключевое слово в центре), в которых перечисляются ключевые слова, обычно выделенные жирным шрифтом в согласованной позиции, в ограниченном количестве контекстного текста, т.е. три [или] четыре слова текста перед ключевым словом и такое же количество текста после него. Этот формат чрезвычайно полезен в том смысле, что ключевое слово легко идентифицируется вместе с его контекстом…. Соответствие Римский Миссал выпускается в форматах KWIC и KWICn и примечателен тем, что каждая словоформа указана в том виде, в котором она представлена ​​в тексте, то есть без лемматизации ».

См. Также

• ptx , служебная программа командной строки Unix, создающая пермутированный индекс
• Concordancer
• Concordance (публикация)
• Преобразование Барроуза – Уиллера
• Ганс Питер Лун
• Суффиксное дерево

Ссылки

July 03, 2019

Contents

• Lesson Goals
• Files Needed For This Lesson
• From Text to N-Grams to KWIC
• From Text to N-grams
• Code Syncing

Lesson Goals

Like in Output Data as HTML File, this lesson takes the frequency
pairs collected in Counting Frequencies and outputs them in HTML.
This time the focus is on keywords in context (KWIC) which creates
n-grams from the original document content – in this case a trial
transcript from the Old Bailey Online. You can use your program to
select a keyword and the computer will output all instances of that
keyword, along with the words to the left and right of it, making it
easy to see at a glance how the keyword is used.

Once the KWICs have been created, they are then wrapped in HTML and sent
to the browser where they can be viewed. This reinforces what was
learned in Output Data as HTML File, opting for a slightly
different output.

At the end of this lesson, you will be able to extract all possible
n-grams from the text. In the next lesson, you will be learn how to
output all of the n-grams of a given keyword in a document downloaded
from the Internet, and display them clearly in your browser window.

Files Needed For This Lesson

• obo.py

If you do not have these files from the previous lesson, you can
download programming-historian-7, a zip file from the previous lesson

From Text to N-Grams to KWIC

Now that you know how to harvest the textual content of a web page
automatically with Python, and have begun to use strings, lists and
dictionaries for text processing, there are many other things that you
can do with the text besides counting frequencies. People who study the
statistical properties of language have found that studying linear
sequences of linguistic units can tell us a lot about a text. These
linear sequences are known as bigrams (2 units), trigrams (3 units), or
more generally as n-grams.

You have probably seen n-grams many times before. They are commonly used
on search results pages to give you a preview of where your keyword
appears in a document and what the surrounding context of the keyword
is. This application of n-grams is known as keywords in context (often
abbreviated as KWIC). For example, if the string in question were “it
was the best of times it was the worst of times it was the age of wisdom
it was the age of foolishness” then a 7-gram for the keyword “wisdom”
would be:

the age of wisdom it was the

An n-gram could contain any type of linguistic unit you like. For
historians you are most likely to use characters as in the bigram “qu”
or words as in the trigram “the dog barked”; however, you could also use
phonemes, syllables, or any number of other units depending on your
research question.

What we’re going to do next is develop the ability to display KWIC for
any keyword in a body of text, showing it in the context of a fixed
number of words on either side. As before, we will wrap the output so
that it can be viewed in Firefox and added easily to Zotero.

From Text to N-grams

Since we want to work with words as opposed to characters or phonemes,
it will be much easier to create n-grams using a list of words rather
than strings. As you already know, Python can easily turn a string into
a list using the split operation. Once split it becomes simple to
retrieve a subsequence of adjacent words in the list by using a slice,
represented as two indexes separated by a colon. This was introduced
when working with strings in Manipulating Strings in Python.

message9 = "Hello World"
message9a = message9[1:8]
print(message9a)
-> ello Wo

However, we can also use this technique to take a predetermined number
of neighbouring words from the list with very little effort. Study the
following examples, which you can try out in a Python Shell.

wordstring = 'it was the best of times it was the worst of times '
wordstring += 'it was the age of wisdom it was the age of foolishness'
wordlist = wordstring.split()

print(wordlist[0:4])
-> ['it', 'was', 'the', 'best']

print(wordlist[0:6])
-> ['it', 'was', 'the', 'best', 'of', 'times']

print(wordlist[6:10])
-> ['it', 'was', 'the', 'worst']

print(wordlist[0:12])
-> ['it', 'was', 'the', 'best', 'of', 'times', 'it', 'was', 'the', 'worst', 'of', 'times']

print(wordlist[:12])
-> ['it', 'was', 'the', 'best', 'of', 'times', 'it', 'was', 'the', 'worst', 'of', 'times']

print(wordlist[12:])
-> ['it', 'was', 'the', 'age', 'of', 'wisdom', 'it', 'was', 'the', 'age', 'of', 'foolishness']

In these examples we have used the slice method to return parts of our
list. Note that there are two sides to the colon in a slice. If the
right of the colon is left blank as in the last example above, the
program knows to automatically continue to the end – in this case, to
the end of the list. The second last example above shows that we can
start at the beginning by leaving the space before the colon empty. This
is a handy shortcut available to keep your code shorter.

You can also use variables to represent the index positions. Used in
conjunction with a for loop, you could easily create every possible
n-gram of your list. The following example returns all 5-grams of our
string from the example above.

i = 0
for items in wordlist:
    print(wordlist[i: i+5])
    i += 1

Keeping with our modular approach, we will create a function and save it
to the obo.py module that can create n-grams for us. Study and type or
copy the following code:

# Given a list of words and a number n, return a list
# of n-grams.

def getNGrams(wordlist, n):
    return [wordlist[i:i+n] for i in range(len(wordlist)-(n-1))]

This function may look a little confusing as there is a lot going on
here in not very much code. It uses a list comprehension to keep the
code compact. The following example does exactly the same thing:

def getNGrams(wordlist, n):
    ngrams = []
    for i in range(len(wordlist)-(n-1)):
        ngrams.append(wordlist[i:i+n])
    return ngrams

Use whichever makes most sense to you.

A concept that may still be confusing to you are the two function
arguments. Notice that our function has two variable names in the
parentheses after its name when we declared it: wordlist, n. These two
variables are the function arguments. When you call (run) this function,
these variables will be used by the function for its solution. Without
these arguments there is not enough information to do the calculations.
In this case, the two pieces of information are the list of words you
want to turn into n-grams (wordlist), and the number of words you want
in each n-gram (n). For the function to work it needs both, so you call
it in like this (save the following as useGetNGrams.py and run):

#useGetNGrams.py

import obo

wordstring = 'it was the best of times it was the worst of times '
wordstring += 'it was the age of wisdom it was the age of foolishness'
allMyWords = wordstring.split()

print(obo.getNGrams(allMyWords, 5))

Notice that the arguments you enter do not have to have the same names
as the arguments named in the function declaration. Python knows to use
allMyWords everywhere in the function that wordlist appears, since this
is given as the first argument. Likewise, all instances of n will be
replaced by the integer 5 in this case. Try changing the 5 to a string,
such as “elephants” and see what happens when you run your program. Note
that because n is being used as an integer, you have to ensure the
argument sent is also an integer. The same is true for strings, floats
or any other variable type sent as an argument.

You can also use a Python shell to play around with the code to get a
better understanding of how it works. Paste the function declaration for
getNGrams (either of the two functions above) into your Python shell.

test1 = 'here are four words'
test2 = 'this test sentence has eight words in it'

getNGrams(test1.split(), 5)
-> []

getNGrams(test2.split(), 5)
-> [['this', 'test', 'sentence', 'has', 'eight'],
['test', 'sentence', 'has', 'eight', 'words'],
['sentence', 'has', 'eight', 'words', 'in'],
['has', 'eight', 'words', 'in', 'it']]

There are two concepts that we see in this example of which you need to
be aware. Firstly, because our function expects a list of words rather
than a string, we have to convert the strings into lists before our
function can handle them. We could have done this by adding another line
of code above the function call, but instead we used the split method
directly in the function argument as a bit of a shortcut.

Secondly, why did the first example return an empty list rather than the
n-grams we were after? In test1, we have tried to ask for an n-gram that
is longer than the number of words in our list. This has resulted in a
blank list. In test2 we have no such problem and get all possible
5-grams for the longer list of words. If you wanted to you could adapt
your function to print a warning message or to return the entire string
instead of an empty list.

We now have a way to extract all possible n-grams from a body of text.
In the next lesson, we can focus our attention on isolating those
n-grams that are of interest to us.

Code Syncing

To follow along with future lessons it is important that you have the
right files and programs in your “programming-historian” directory. At
the end of each chapter you can download the “programming-historian” zip
file to make sure you have the correct code. If you are following along
with the Mac / Linux version you may have to open the obo.py file and
change “file:///Users/username/Desktop/programming-historian/” to the
path to the directory on your own computer.

• python-lessons8.py (zip sync)