Question and answer word search

 

7.0 Overview

In January 2011, IBM shocked the world by beating two former Jeopardy! champions with its Watson DeepQA system. The Watson victory was a prominent feature in press reports and the subject of a popular book (Baker, 2013).

This chapter starts by explaining how the IBM Watson system worked. Though it was far from a human-level natural language processing system, it was an amazing accomplishment and is a great example available of how clever programming can make a system appear to be intelligent when nothing could be further from the truth.

Following the discussion of Watson will be a discussion of question-answering research that has taken place since Watson and a description of how AI researchers have attacked the problem of question answering.  Question answering research can be divided into five research categories:

(1) Text-to-SQL

Corporations and other organizations have massive amounts of data stored in relational databases.  Text-to-SQL systems translate natural language requests for information into SQL commands.

(2) Knowledge Base QA

Knowledge bases store factual information.  KB-QA systems translate take natural language factual questions as input and retrieve facts from KBs in response.

(3) Text QA

Text QA systems take natural language questions as input and search document collections such as Wikipedia to find answers.  In 2018, both Microsoft and Alibaba announced they had built systems that matched human performance on a standardized reading comprehension test. However, as will be explained below, this reading comprehension test is susceptible to clever Watson-like tricks, and the systems that do well on this and similar tests use these tricks and really don’t understand language like people understand language.

(4) Task-oriented personal assistants and social chatbots

Like KB-QA and text-QA systems, personal assistants (e.g. Alexa and Siri) respond to natural language questions.  Additionally, they respond to commands and also keep track of the dialogue state in an effort to provide responses that make sense given the history of the dialogue.  Academic research on these types of systems is termed Conversational QA (Zaib et al, 2021).

(5) Commonsense reasoning QA

As will be discussed below, many questions can be answered without requiring any high-level reasoning.  Commonsense QA datasets are designed with questions that are difficult to answer without knowledge of the real world and the ability to reason about that knowledge.

This chapter will discuss Text-to-SQL systems, KB-QA, and Text-QA.  The chatbots chapter will discuss task-oriented personal assistants and social chatbots.  

7.1 IBM’s Watson question answering system

Jeopardy! is a popular question-answering TV show in which a contestant picks one of six categories. Then all three contestants are presented with a question from that category. The first contestant to hit a buzzer is given the opportunity to answer the question. (Actually, the question is given in the form of an answer and the answer must be given in the form of a question). There is a significant penalty for wrong answers. All questions must be answered from “memory”. And no access to the internet is allowed.

PIQUANT: Watson’s question answering predecessor

IBM’s Watson question answering system was developed as a follow-on to research on a question-answering system named PIQUANT that was funded by government research grants and that performed well in conference competitions between 1999 and 2005 (Ferrucci, 2012). The PIQUANT system was oriented toward answering simple factual questions of the type that can be answered using a knowledge base.

The IBM Watson question answering project was kicked off in April 2007 and culminated with the stunning victory in January 2011. The team first considered building on PIQUANT. However, after analyzing the performance of prior Jeopardy! winners, the team determined that Watson would need to buzz in for at least 70% of the questions and get 85% correct. As a baseline, they tested the PIQUANT system and found that when it buzzed in 70% of the time, it only got 16% correct.

The basic approach of PIQUANT was to first determine the knowledge base category of the question. A specialized set of handcrafted information extraction rules were created for each high-level category (e.g. people, organizations, and animals) that found the correct instance of the type. This was possible because of the constrained domain of the conference tasks.

That approach doesn’t work for Jeopardy! or for open-domain question answering (i.e. not restricted to a small subject area) in general because there are just too many possible entity types. For example, the YAGO ontology has 350,000 entity types (i.e. classes). Developing 350,000 separate algorithms would be a big effort.

Documents stored in the IBM Watson system

The Jeopardy! rules do not allow contestants to access the internet, so part of the challenge was to collect and organize a huge body of information. Wikipedia was the primary information source as more than 95.47% of Jeopardy! questions can be answered with Wikipedia titles. The IBM Watson team (Carroll et al, 2012) cited this example:

TENNIS: The first U.S. men’s national singles championship, played right here in 1881, evolved into this New York City tournament. (Answer: US Open)

This clue contains 3 facts about the US Open (1^st US men’s championship, first played in 1881, now played in New York City) that are all found in the Wikipedia article titled “US Open (tennis)”. The IBM Watson team noted that Wikipedia articles are also key resources when an entity is mentioned in the question (answer). For example:

Aleksander Kwasniewski became the president of this country in 1995. (Answer: Poland)

The second sentence of Kwasniewski’s Wikipedia article says:

He served as the President of Poland from 1995-2005.

The IBM team then analyzed the 4.53% of questions that couldn’t be answered via Wikipedia titles and found several types of questions. Many questions involved dictionary definitions so they incorporated a modified version of Wiktionary (a crowd-sourced dictionary) which, like Wikipedia, was already title-oriented. Other common sources of questions included Bible quotes, song lyrics, quotes, and book and movie plots and they created title-oriented documents for these categories.

For example, for song lyrics, they created one set of documents with the artist as the title and the lyrics in the article and another set with the song title as the title. As will be explained below, the IBM Watson system used many different strategies for matching questions to documents. Nearly all matching algorithms were lexically-oriented. In other words, the matching algorithms matched words in the questions to words in the documents rather than extracting the meaning of the question into a canonical meaning representation, extracting the meaning of the document into a canonical meaning representation, and matching on the meanings rather than the words.

Since there are many different word combinations for expressing most facts and since each Wikipedia and/or Wiktionary article typically has only a single lexical form for each fact, the team augmented popular articles and definitions with other text found on the web and news wire articles in the hope of capturing other lexical forms of important facts and increasing the likelihood of matching question words or synonyms to documents words or synonyms. These document acquisition methodologies expanded the initial Wikipedia-based knowledge-base from 13 to 59 GB and improved Jeopardy! performance by 11.4%.

Information extraction in IBM Watson

Identifying entities and relations in questions and linking them to Wikipedia titles and other stored documents was a key capability for the IBM Watson DeepQA system.

It used an open-source natural language processing framework that was developed from 2001 to 2006 by IBM named the Unstructured Information Management Architecture (UIMA). UIMA includes tools for language identification, language-specific segmentation, POS tagging, text indexing, sentence boundary detection, tokenization, stemming, regular expressions, and entity detection.

The IBM Watson DeepQA system also used a parsing and representational system named slot grammars (McCord et al, 2012) that captures both surface linguistic information (e.g. POS), morphological information (e.g. tense and number), logical argument structure (subject, object, complement), and dependency information. English, German, French, Spanish, Italian, and Portuguese implementations of the parser were used.

The DeepQA system automatically extracted a set of entities and relations from Wikipedia and used them to create a lexicon that could drive recognition of entities and relations in Jeopardy! questions. To learn lexical entries (i.e. words or phrases) that can be used to identify Wikipedia entities, DeepQA didn’t simply rely on keyword matching to Wikipedia titles. In addition to keywords, they used link information (i.e. words in Wikipedia articles that are hyperlinked to the entity article) to match different surface forms (i.e. words and phrases) to each entity in Wikipedia.

The extracted lexicon included 1.4 million proper noun phrasal entries extracted from Wikipedia, phrasal medical entries extracted from the Unified Medical Language System, and numerous entity and type synonyms extracted using the relationships between Wikipedia, YAGO, and WordNet.

To learn a set of relations and a lexicon to identify each relation, the IBM Watson DeepQA system used Wikipedia infoboxes which frequently contain relations. For example, in describing the IBM Watson relation extraction process, the IBM Watson authors (Wang et al, 2011) note that the infobox in the article for Albert Einstein lists his “alma mater” as “University of Zurich”. Then the first sentence in the article mentioning “alma mater” and “University of Zurich” says

Einstein was awarded a PhD by the University of Zurich

Sentences like this were then used as training sentences for their relations. Classifiers were automatically trained for over 7000 relations based on this type of analysis of Wikipedia text. The IBM Watson team also defined 30 handcrafted patterns to identify common relations with high confidence. An example pattern to identify the authorOf relationship was

[Author] [WriteVerb] [Work]

They also developed a “true case” algorithm to account for the fact that clues are presented in all upper case. This algorithm was trained on thousands of same phrases to transform the upper case clues into mixed case clues and thereby provide algorithms (e.g. NER) with additional information.

How IBM Watson interpreted the questions

When the IBM Watson DeepQA system read a Jeopardy! question, it extracted four key pieces of Jeopardy!-specific information:

Focus: The focus was determined by a set of handcrafted rules such as

A noun phrase with determiner “this” or “these”.

This rule would pick up the correct focus in question such as

THEATRE: A new play based on this Sir Arthur Conan Doyle canine classic opened on the London stage in 2007.

If none of the rules matched, the question (answer) was treated as if it didn’t have a focus.

Lexical answer types (LATs): The DeepQA interprets each question as having one or more Lexical Answer Types (LATs). A LAT is often the Focus of the question. It can also be a class. For example, “dog” would be a LAT in the question

Marmaduke is this breed of dog.

The set of LATs can also include the answer category. For example, “capital” will be a LAT for questions in the category

FOLLOW THE LEADER TO THE CAPITAL

You can think of the LAT as a type of concept. For example, a clue might indicate the response is a type of dog. In this case, “dog” would be the LAT. Here also, we can run into a word matching problem. A clue might reference the concept associated with the word “dog” by many different words and phrases, including “dog,” “canine,” and “man’s best friend.”

Fortunately, there are ontologies that provide this type of mapping. An open-source ontology named YAGO is available online that maps words to concepts, and the IBM Watson team used YAGO to map the LAT word(s) onto YAGO concepts. There is also an open-source mapping from Wikipedia to YAGO named DBPedia, and DeepQA used this to map possible answers to YAGO concepts.

YAGO also contains hierarchical information about concepts. For example, YAGO lists an animal hierarchy (animal->mammal->dog). When the entity and LAT were both available in YAGO, DeepQA used conventionally coded hierarchical rules to determine if the entity matched the LAT concept exactly or matched a subtype or supertype in the YAGO concept hierarchy.

Certain LATs (e.g., countries, US presidents) have a finite list, and the list instances often are matched to the entities in candidate answer documents using Wikipedia lists.

See the IBM Watson team paper on this topic (Lally et al, 2012) for information on how DeepQA determines the LATs.

Question Classification: Factoid questions are the most common type of Jeopardy! question. However, the IBM team found 12 other question categories (e.g. Definition, Multiple-Choice, Puzzle, Common Bonds, Fill-in-the-Blanks, Abbreviation). They developed a separate set of handcrafted rules for recognizing the question class and applied different answering techniques for each type.

Question Sections (QSections): These are annotations placed by the IBM Watson system on the query text that indicate special treatment is required. For example, a phrase like

…this 4-letter word…

is marked as a LEXICAL CONSTRAINT which indicates that it should be used to select the answer but should not be part of the answer. Like Question Classes, these are identified by a set of handcrafted rules.

Using the parsing information discussed earlier, the entities and relations are also extracted from the question.

How IBM Watson identified the candidate answers

The next step is to identify Wikipedia and other documents that contain candidate answers. Consider, for example, the following clue (Carroll et al, 2012b).

MOVIE-ING: Robert Redford and Paul Newman starred in this depression-era grifter flick.

(Answer: The Sting)

Both the focus and LAT are “flick” and two relations are extracted:

actorIn (Robert Redford; flickflick : focus)

actorIn (Paul Newman; flick : focus)

Four types of document searches are performed:

Titled document search: Titled documents (e.g. Wikipedia and Wiktionary) are searched for “Robert Redford”, “Paul Newman”, “star”, “depression”, “era”, “grifter”, and “flick” and rated by several criteria. For example, documents with both the Redford and Newman names are weighted the highest. Documents with the focus word “flick” are weighted next highest. The 50 long documents (e.g. Wikipedia) and 5 short documents (e.g. Wiktionary) that are rated highest are considered candidates. The document titles become the candidate answers.

Passage search: Untitled documents (e.g. news articles) are searched using the same terms in two stages to identify one- or two-sentence passages that contain candidate answers.

First, articles are found for which the words in the headline is contained in the clue. The ten highest-rated candidate passages are retrieved.

Second, all other articles are searched using two different search engines with very different algorithms. The documents are rated based on the similarity of the words in the passage to the words in the query and other factors such as whether the sentence is close to the beginning of the article. Similarity includes exact word matches plus synonym and/or thesaurus matches.

The five highest-rated passages for each search engine are considered candidate documents. For each candidate document, a variety of lexically-oriented techniques are used to extract the candidate answer (Ferrucci et al, 2012).

Structured databases: One might think that simply looking up answers in structured databases such as IMDB (Internet Movie Database) and/or a knowledge base (DBPedia was the knowledge base used by DeepQA) would provide an answer to most factoid questions. The difficulty is that this requires matching the words in the names used in the query to the words in the names in these structured data sources. Moreover, it relies on the words identifying the relations discovered matching the words defining the relations in these data sources. Said another way, the relations might match, but if the words used to express the relation in the question and database are different, the system won’t see the match. As a result, while this technique was found to be effective for some types of relations, especially the ones extracted with handcrafted rules such as actorIn, it was generally a less effective strategy than Title Document Search and Passage Search.

PRISMATIC knowledge base: The IBM Watson team created its own knowledge base named PRISMATIC (Fan et al, 2012). The facts in PRISMATIC are extracted from encyclopedias, dictionaries, news articles, and other sources. The facts include commonsense knowledge such as “scientists publish papers” that can be used to help interpret questions. DeepQA used PRISMATIC as another source of candidate answers.

How IBM Watson scores the candidate answers on LAT match

One of the most important steps in scoring each candidate answer is to determine the likelihood that it matches the LAT. To accomplish this, the IBM watson system canonicalizes entity references using several different techniques including:

• The NER detector from PIQUANT which is used for 100 common named entity types.
• Entity linking by matching the entity mention string to strings in Wikipedia articles that have hyperlinks. Different ways of mentioning the entity in Wikipedia are all linked back to the same Wikipedia page thereby providing a means of canonicalizing the lexical variation in entity references.
• Word co-occurrence algorithms using Wikipedia disambiguation pages and other unstructured and structured resources.

Next, the canonicalized entity reference is matched to the LAT. However, the LAT is just a word or string of words and it also needs to be canonicalized by finding its underlying concept in an ontology. IBM Watson system uses the YAGO ontology and uses WordNet / YAGO links to map words in the mention string to concepts in YAGO. The LAT mention string is looked up in WordNet. In some cases, there are multiple word senses.

The IBM Watson team (Kalyanpur et al, 2012) offered the example of the word “star”. This has two WordNet word senses. The higher ranked word sense is the astronomy sense. The lower-ranked word sense is the movies sense. As a result, the LAT will be associated with two concepts. It will be associated with the astronomy sense with a 75% probability and the movie star sense with a 25% probability. However, if the candidate answer has a named entity that is a movie star, the match will still occur.

The relative probabilities are then adjusted by determining which word senses have more instances in DBPedia and are therefore presumed to be more popular. The IBM Watson system uses several methods to assess each candidate answer against each LAT (Murdoch et al, 2012). Some of these methods are:

Wikipedia Lookup: The type of the entity concept extracted from the candidate answer is looked up in YAGO and compared with the YAGO LAT concept. For example, if the candidate answer matches a link to a Wikipedia title, the entity is looked up in DBPedia which has a cross-reference to YAGO types. Wikipedia articles are also commonly tagged with categories. These categories are matched against the LAT.

Taxonomic Reasoning: When the entity and LAT can both be found in YAGO, it is possible to do taxonomic reasoning and determine if the entity matches the LAT concept exactly or matches a subtype or supertype in the YAGO concept hierarchy.

Gender: A special algorithm is used to compare LAT’s that specify a gender (e.g. “he”) to an entity in the candidate answer.

Closed LATs: Certain LATs (e.g. countries, US presidents) have a finite list and the list instances are matched to the candidate entity.

WordNet: The lexical unit hierarchy is checked for matches between the LAT and candidate answers.

Wikipedia Lists: Wikipedia contains many lists. These lists are stored and matched against question text starting with “List of…”.

Each method also assigns a confidence rating to the type match.

Additional evidence used by IBM Watson

Additional evidence is also gathered for each candidate answer. For example, each candidate answer is merged into the question, and searches are performed looking for text passages that contain some or all the question words in any order. Over 50 separate scoring algorithms are used to create a set of scores that are then used as features in a machine learning algorithm. Some examples:

• The found passages are scored by several different scorers that analyze things like how many question words are matched and how well the word orders match.
• There are multiple temporal answer scorers that determine if the dates for the candidate answer are consistent with the dates in the question. If the question asks about the 1900s, someone who lived in the 1800s is not a plausible answer. Temporal references are normalized and matched against DBPedia which has dates (e.g. birth/death dates of people) and durations of events. Temporal data is also extracted into a DeepQA knowledge base that contains the number of times a date range was mentioned for an entity and this knowledge base is used to score time period references in candidate answers.
• Similarly, location scorers assess the geographic compatibility between the question and candidate answer. Longitude/latitude specifications in a query are normalized using regular expressions and matched against location data stored in DBPedia and Freebase.
• Other scorers check handcrafted rules such as an entity can’t be both a country and a person. There are over 200 such rules.

How IBM Watson computed the final scores

Candidate answers are ultimately scored by applying a logistic regression algorithm (with regularization) to rank candidate answers according to a set of 550 features (Gondek et al, 2012). The classifier is trained to compare each pair of candidate answers and identify the best answer. This produces a ranking of the answers.

A supervised learning algorithm was trained on 25,000 Jeopardy! questions with 5.7 million question-answer pairs (including both correct and incorrect answers). The answer produced by the IBM Watson system is the candidate response with the high rank (converted into a question).

Summary of IBM Watson question answering

The IBM Watson system is an amazing feat of engineering. The result is a system that appears to understand complex English questions. Under the hood, what is really occurring is a massive set of mostly handcrafted rules that extract entities and relations, a set of rules for keyword matches between words in the questions and words in the stored documents and databases, and a massive set of handcrafted rules for identifying the most likely entity answer from a large set of candidate answers.

There is no attempt by the IBM Watson system to understand a question by producing a meaning representation of the question and matching that meaning representation to the meaning representations of documents and knowledge bases. As David Ferrucci, the IBM Watson team leader, said (Ferrucci, 2012),

“DeepQA does not assume that it ever completely understands the question…”

Most of the system is based on brute force lexical matching of questions to documents. The rules used to match the time frame and location specified in the question to the time frame and location specified in the candidate answers and to match the entities and relations in the question to the candidate answers could be considered a form of reasoning if one stretches the definition. However, clever programming is a more apt description of the entire system.

7.2  Text-to-SQL

One of the major early areas of focus of both academic research and commercial system was in the area of natural language interfaces to SQL databases.  In the 1970s and 1980s, before business intelligence software revolutionized the way knowledge workers interact with databases, it was thought that natural language interfaces had huge potential for bridging the gap between non-technical business people and the information in structured databases.

Starting in the 1980s, several commercial text-to-SQL systems were created with varying degrees of success.  All of these commercial systems used symbolic NLP techniques.  Most current research uses machine learning.  Both will be discussed below.

7.2.1  Text-to-SQL challenges

Text-to-SQL systems face four major challenges:

(1) Schema encoding

Text-to-SQL systems need to translate natural language into a representation that capture the database schema semantics, i.e. the database tables, columns, and the relationships between the tables.  If the text-to-SQL system requires significant manual effort for each new database, the implementation cost can be a barrier to the use of the text-to-SQL system.

(2) Schema linking

Text-to-SQL systems need to correctly interpret references to the tables and columns of the database in the query.  Here also, manual effort to create the lexical entries for each new databases is a barrier to use.

(3) Multi-table schemas

Most real-world databases have multiple tables and text-to-SQL systems need to generate SQL that includes joins to link these tables.  Text-to-SQL systems developed for single-table databases won’t have this capability.

(4) Syntactically correct but semantically incorrect queries

Many queries contain syntactically correct SQL and, when run against the database, will produce output.  However, the output can be meaningless.  One example, is SQL with many-to-one-to-many joins.  For example, a database may have a table of payments by customer and another table of orders by customer.  However, SQL that merges the data from these two tables will generate incorrect data.  The data will look like it shows payments for specific orders but that information is not actually contained in the tables.

7.2.2 Datasets for training text-to-SQL systems

Many datasets are available for research on natural language interfaces to databases. These datasets include a SQL database and include questions and correct SQL broken down into training and test sets. Most of these datasets contain databases with a single table. These include

• ATIS (Hemphill et al, 1990), a flight database
• Restaurants (Tang and Mooney, 2000), a database of restaurant locations and food types
• GeoQuery (Zelle and Mooney, 1996), a database of states, cities, and other places
• Academic (Li and Jagadish, 2014), a database of academic publications
• Scholar (Iyer et al, 2017), another database of academic publications
• Yelp, a dataset of Yelp data
• IMDB, a movie database
• Advising (Finegan-Dollak et al, 2017), a database of course information

More recently, researchers have developed datasets that cross a wide range of topics. These include

• WikiSQL (Zhong et al, 2017), a database of questions and answers from a wide range of Wikipedia articles so topic identification is important. Each question references a single table so there is no need for joins.  The WikiSQL leaderboard can be found here.
• Spider (Yu et al, 2018), a database of questions and answers cover a wide range of databases extracted from college database classes. Many questions require joins between tables.  The Spider leaderboard can be found here.

More importantly, the articles/databases in these last two datasets are not present in the training sets. This forces the system to learn generalized patterns, not ones that are specific to the training data.

7.2.3 Symbolic NLP approaches

The first text-to-SQL system was developed in 1960 by MIT researcher Bert Green (who became my Ph.D. thesis advisor 18 years later).  Green and colleagues wrote a computer program (Green et al, 1961) that could answer questions about the 1959 baseball season like:

Who did the Red Sox lose to on July 5? Where did each team play on July 7? What teams won 10 games in July? Did every team play at least once in each park?

Similarly, the LUNAR system (Woods et al, 1972) allowed geologists to ask questions such as:

Give me all lunar samples with magnetite. In which samples has apatite been identified? What is the specific activity of A126 in soil? Analyses of strontium in plagioclase. What are the plag analyses for breccias? What is the average concentration of olivine in breccias? What is the average age of the basalts? What is the average potassiudrubidium ratio in basalts? In which breccias is the average concentration of titanium greater than 6 percent?

In 1971, the LUNAR system was put to a test by actual geologists. Of 111 moon rock questions posed by geologists, 78% were answered correctly. In 1978, Stanford Research Institute developed a system named LADDER (Hendrix et al, 1978) that retrieved Navy ship data.

In 1981, Wendy Lehnert and I (Shwartz, 1982; Lehnert and Shwartz, 1983) developed a natural language interface to oil company well data where one could ask questions like

Show me a map of all tight wells drilled before May I, 1980 but since May I, 1970 by texaco that show oil deeper than 2000, were themselves deeper than 5000, are now operated by shell, are wildcat wells where the operator reported a drilling problem, and have mechanical logs, drill stem tests, and a commercial oil analysis, that were drilled within the area defined by latitude 30 deg 20 min 30 sec to 31:20:30 and 80-81. scale 2000 feet.

These systems all used semantic parsing.  Semantic parsers have hand-coded rules that translate natural language queries into a very narrow frame-based meaning representation that only has slots for the information necessary to create a computer program to retrieve the data. These slots primarily fell into three categories:

• A list of fields and/or calculations to be retrieved
• A set of filters that restricted the information
• A set of sorts that defined how the output should be sorted

For example, the EasyTalk commercial text-to-SQL system (Shwartz, 1987) would translate inputs such as

Show me the customer, city, and dollar amount of Q1 sales of men’s blue shirts in the western region sorted by state.

This would result in a frame-based representation:

Display:

customer_name customer_city customer_state sales_$

Filters:

transaction_date >= ‘1/1/1983’ transaction date <= ‘3/31/1983’ product_id in (‘23423’, ‘23424’, ‘24356’) region = ‘west’

Sorts:

customer_state

In order to parse natural language inputs into this frame-based representation, the EasyTalk system used a hand-coded lexicon. A lexicon contains entries for individual words and tokens, phrases, and/or patterns.

So, for example, the word “customer” by itself and the phrase “customer name” both map to the customer_name field. The token “Q1” maps to a transaction date filter. And so on.

From this internal representation, it is a simple matter to generate SQL or some other database query language to retrieve and display the requested data. For example, the Display elements become the SELECT parameters, the Filters become the WHERE clause, and so on.

It was also a simple matter to create paraphrases of the information in order to confirm with the user that the system correctly understood the request. Early natural language interfaces to databases had hand-coded lexicons some of which enabled inferences such as inferring from

Who owes me for over 60 days?

that an outstanding balance is being requested.

However, the amount of hand-coding for each new database and the required expertise made widespread development of this type of interface impractical.

In 1983, Barbara Grosz at the Stanford Research Institute, developed a framework for TEAM a “transportable” natural language database interface. General domain-independent language patterns for identifying the columns to be retrieved and the filters to be applied were hand-coded. File, record, and field names were retrieved directly from the database and users could enter synonyms for these items. The result was a model for a system that could be applied to any database. The domain specifics were just synonyms of field names that didn’t require special linguistics expertise.

That same general idea was adopted by my EasyTalk system and by several other commercial companies. I started one of those other companies where we produced the Esperant system. Another commercial product is EasyAsk which is a text-to-SQL system that evolved from early research at Dartmouth College (Harris, 1978). Microsoft developed PowerBI Q&A.  Tableau offers Ask Data.  And there are many more.

Some of these commercial systems improved on the transportability idea via additional mechanisms such as reading description fields and automatically creating lexicon entries based on those fields. For example, a product description field might contain “blue men’s medium long-sleeved shirt”. From that description, the EasyTalk system would automatically generate patterns that would match “blue shirt”, “men’s long-sleeved blue shirt”, and so on.

The market need for text-to-SQL systems diminished with the advent of graphical business intelligence tools in the early 1990s. The problem was that, as user interface pioneer Ben Schneiderman pointed out (Schneiderman, 1993), people would rather use a reliable and predictable user interface than an intelligent but unreliable one.

A group of University of Washington researchers (Popescu et al, 2003) argued that, to guarantee reliability, natural language inputs had to be restricted to those that contain unambiguous references to database elements.

Commercial natural language interfaces to databasestext-to-SQL system are far from 100% reliable unless users impose their own constraints on the allowable natural language inputs by learning what questions they can ask.

Interestingly, less than 100% reliability has more recently become acceptable as evidenced by the widespread popularity of chatbots which also require users to either learn what they ask or accept much less than 100% reliable responses.

My own efforts moved in the direction of graphical interfaces with the development of Esperant which was one of the leading business intelligence products of the early and mid-1990s.  In addition to accepting free-form natural language input, Esperant led the user through a set of menu-based choices that led to the construction of a natural language sentence that was guaranteed to produce a 100% reliable answer. For queries that required information from multiple tables, Esperant would gray out choices that would lead to incorrect joins.

7.2.4 Machine learning text-to-SQL approaches

Researchers have been studying the use of machine learning to create natural language interfaces for over 20 years (e.g. Wong and Mooney, 2006; Zettlemoyer and Collins, 2007). 

For example, Zettlemoyer and Collins trained a system to learn a lexicon and grammar for translating English queries into database retrieval requests. They started with a dataset of queries annotated with the formal meaning representation for each query. An example query (Zelle and Mooney, 1996) was

What is the capital of the state with the largest population?

The meaning representation for that query was

answer(C, (capital(S,C), largest(P, (state(S), population(S,P)))))

The system used a set of handcrafted features and a supervised learning algorithm to learn a lexicon and grammar based on a set of lexical features in the queries.  One significant limitation of this approach was the degree of effort required to create the dataset of annotated queries for each database.

The approach has also been used only for relatively small databases such as Geobase which contains 800 facts (e.g. population, neighboring states, area, capital city, major cities, major rivers, and elevation) about US states.

More recently, sequence-to-sequence neural networks have been used for this task. Dong and Lapata (2016) used an attention-based encoder-decoder model to translate from natural language questions to any arbitrary meaning representation or logical form with a formal grammar. They demonstrated its efficacy on several databases including Geobase.

Salesforce (Zhong et al, 2017) and UC Berkeley (Xu et al, 2017) researchers applied a seq2seq model directly to the problem of generating SQL from natural language inputs. They created the WikiSQL dataset. Their model generates a SQL query that gets the correct answer about 70% of the time. However, they can only do this for a single table (i.e. no joins) and the questions are limited to those that contain a column name.

More recently, researchers have created schemes that enable a system to learn information about the database schema and relate it to the question. Allen Institute researchers (Bogin et al, 2019) used a graph neural network to learn the schema information and used an encoder-decoder to convert text to SQL.  They used the Spider dataset and their system was able to create joins and complex SQL constructs including subqueries.

Microsoft researchers (Wang et al, 2020) increased performance on the Spider dataset by creating a system named RAT-SQL using relation-aware self-attention in which they encode the schema foreign key information and the questions in a common format. This allows using a transformer architecture to jointly learn questions and database schema, to generate SQL that contains joins, and to restrict the generated SQL to syntactically correct SQL (but not necessarily semantically correct).

As of June 2022, the leading system with published data on the 

dataset was PICARD (Scholak et al, 2021).  At each step of the decoding process, pre-trained language models can produce 10,000+ sub-word tokens, some of which are grammatically incorrect.  PICARD rejects these inadmissable subword tokens at each step of the generation process.

7.2.5 Text-to-SQL  summary

In many NLP sub-disciplines, machine learning approaches are outpacing symbolic approaches.  The opposite is true for text-to-SQL systems.  Commercial systems using symbolic NLP have long been able to produce joins (i.e. answer natural language queries involving more than one table) and recognize the column name if an unordered subset of a column value (e.g. “blue shirts”) is used in the question.  These capabilities are created using automated tools and do not require manual coding and have been available for 35 years (Shwartz, 1987). 

Additionally, symbolic text-to-SQL systems have the ability to generate SQL that is both syntactically AND semantically correct.  Machine learning researchers have not yet begun to study this issue.

7.3 Knowledge base QA

Recall that knowledge bases (KBs) like Wikidata store facts like (Obama, BornIn, Hawaii). Queries against KBs like Freebase are considered open domain because of the high breadth of stored knowledge and queries against Wikipedia and other text sources have even a higher breadth of stored knowledge.  In constrast, closed domain QA systems operate in a single domain such as restaurants or movies.

Queries against KBs offer the opportunity to develop a single system that can answer a broad range of questions against a wide variety of topics. The two primary approaches to KBQA that will be discussed are semantic parsing and information retrieval. 

7.3.1 Semantic parsing

One approach is to perform a syntactic and semantic analysis of the question to parse it into a meaning representation in which the entities are not yet linked to the KB.  Then the entities are mapped to entities in the KB and the meaning representation is converted to a logical that can be executed as a KB query. For example, two Temple University researchers (Cai and Yates, 2013) studied learning algorithms for the more than 2000 relations in the Freebase KB. KB’s contain simple facts that can be used to answer a wide variety of simple questions such as:

What are the neighborhoods in New York City?

How many countries use the rupee?

Question (1) asks about the relation “neighborhoods” and the entity “New York City”. Question (2) asks about the relation “countries_used” and the entity “rupee”. However, if we strip out the relation and entity terms, the questions are much more generic:

What are the <relation> in <entity>?

How many <relation> the <entity>?

One can think of the learning problem as one of trying to learn all the different general forms of queries like (3) and (4) and learning to recognize the words used to identify the relations and entities in the KB.

The Temple researchers reasoned that a supervised learning algorithm on its own would require an enormous dataset that contained annotated questions most combinations of question type, relation, and entity. They tested this idea by using a supervised learning algorithm against a training dataset that had one set of relations and entities and found that the system could not answer a single question from a dataset of questions with a different set of entities and relations.

Instead, they used a supervised algorithm to essentially learn the different query types and a separate algorithm to learn the different ways of expressing the KB relations and entities in natural language texts. Their approach to learning words that reference relations and entities was quite clever and borrows techniques from the discipline of schema matching.

7.3.2 Information retrieval

Information retrieval approaches do not produce an intermediate logical form and instead map directly from the question to the structure of the KB.  They typically try to match a representation of the query to multiple candidate subgraphs from the KB.  For simple queries that match a single KB fact, the answer is the best-matching candidate subgraph.  For complex queries, an intermediate logical reasoning step is required (see next section).

Early information retrieval approaches relied on handcrafted rules.  For example, University of Washington researchers (Fader et al, 2014) developed an open question answering system named OQA for simultaneously querying a set of KBs. Rather than learning to map natural language questions for logical forms, ten handcrafted regular expression templates were used. An example of one of the ten templates is:

Template: Who/What RV_rel NP_arg (?x, rel, arg) Example Question: Who invented papyrus? Example Query: (?x, invented, papyrus)

RV stands for relational verbs identified by an open information extraction system. NP stands for noun phrase. The resulting query is then used to search the four KBs for matching triples. If this were the end of the story, the resulting system wouldn’t be very successful because the relation words used in the question might not match the relation words used in the KB. For example, a KB might contain the fact

(papyrus, was invented by, ancient egyptians)

which does not match the query

(?x, invented, papyrus)

Because of the dependence on surface lexical forms, the system needs the following query rewrite operator:

(?x, invented, ?y) (?y, was invented by, ?x)

The system learned 74 million rewrite operators by searching the KBs for pairs of triples in which the two entities match. For example, these two triples were found:

(papyrus, was invented by, ancient egyptians) (ancient egyptians, invented, papyrus)

Because the relations “invented” and “was born to” were found with at least 10 other entity pairs, this rewrite rule was learned:

(?x, invented, ?y) (?y, was invented by, ?x)

The system also learned 5 million paraphrase rules applied directly to the lexical forms of question by analyzing 23 million sets of sentences that were listed as synonymous in a large crowdsourced dataset of questions and answers formerly named WikiAnswers. An example lexical rewrite rule is:

What is the latin name for …? What is …’s scientific name?

The Open QA system was also able to process queries asking for lists and counts instead of just individual facts and could handle conjunctions (ANDs and ORs).

The advantage of this approach is that it does not require handcrafted KBs and/or annotated datasets and is, therefore, more likely to be applicable to many different types of KBs, specialized vocabularies, and languages other than English.

A more recent version of this open domain question answering system is available for public use at openie.allenai.org. Open QA techniques are lexically-based. Words, phrases, and synonyms in questions are matched to words, phrases, and synonyms in knowledge base entries and there is no real understanding of the meaning of the questions.

More recently, information retrieval approaches to KBQA have relied on deep learning.  Some systems, termed memory networks, learn embeddings for both the questions and KB facts and match the similarity of the embeddings to find the best match for a question either directly (Bordes et al, 2015) or using an attention mechanism (Sukhbaatar et al, 2015).

A group of Facebook researchers (Weston et al, 2015) developed memory networks to compensate for the fact that RNN networks (including LSTMs) do not have a distinct memory other than what is compressed and encoded into the system weights. As a result, RNNs are known to have difficulty in performing tasks that would require memory in people. For example, RNNs cannot perform memorization, the simple copying task of outputting the same input sequence they have just read (Zaremba & Sutskever, 2015). To remedy this deficiency, they developed a neural network augmented with an explicit memory. A memory network processes each question in four steps:

1. Input Module: Converts the question string to an input feature representation.
2. Memory Module: The memory is updated given the new question string.
3. Output Module: A set of output features are computed given the new input feature representation and the current state of the memory.
4. Response Module: The output features are decoded to produce an answer.

For example, Bordes et al (2015) used a memory network to answer questions against the WebQuestions dataset (Berant et al, 2013).

Both the KB itself and the training questions were processed by the input module. Each question in the training set was converted into a set of input features composed of a bag-of-ngrams representation of words in the questions. A vector was created of all the words in all the training questions plus all the n-grams (i.e. phrases) for all the aliases of all the Freebase entities.

For each question, the vector contained a 1 if the word or n-gram was present in the question and a 0 if not present. Each fact in the subset of the Freebase KB used was also loaded into a vector representation. In this case, the vector had an entry for each entity, and each relationship in the subset of the KB that is part of the fact triple. Two embedding matrices are learned, one for the words and n-grams in the questions and one for the entities and relationships that make up the facts. At each step of the learning process, the following occurs:

1. A question along with the answer fact are sampled
2. An incorrect answer fact is randomly retrieved
3. A stochastic gradient descent step adjusts the weights according to a cost function that requires that correct facts are preferred over incorrect facts.
4. The output module finds the best match between a new question and an existing fact during both training and test.

7.3.3 Complex KBQA

The KBQA approaches just discussed are applicable only to questions that can be answered by finding a single relation in the KB. For example, the question

What films has Harrison Ford acted in?

can be answered by a single relation in a movie KB that has an acted_films relation. But many KB questions cannot be answered by directly matching a single relation to the query.

Complex KBQA involves questions that require one or more of the following:

1. Multi-hop questions that require accessing two or more different KB relations
2. Questions that require simple logical reasoning
3. Questions that require numerical operations such as counting or ranking for superlative or comparative questions
4. Questions that require parsing constraints such as time constraints (e.g. “in the 1800s”) and/or constraints on entities (e.g. “former CEO Lewin”)

Complex queries make semantic parsing more difficult because the logical forms are more complex.  Moreover, if supervised learning is contemplated, logical form labeling becomes more difficult, more expensive, and requires higher-skilled annotators. 

Complex queries also make IR approaches difficult because it is no longer possible to assume that the entities and relation in the query correspond to a single fact in the KB.  If supervised learning is contemplated, it is often helpful to annotate the learning path but this is far more difficult and expensive with complex queries.

7.3.3.1 Multi-hop KBQA

A question like

What films has Harrison Ford acted in that were produced by George Lucas?

would require accessing two relations (e.g. acted_films and produced_films) and taking the intersection of the results. Knowledge base question answering systems perform poorly on questions that require multiple relations. Performance degrades by approximately 50% when four KB relations are required to produce an answer versus when only one KB relation is required (Sorokin and Gurevych, 2018). These systems also cannot handle questions that require reasoning based on time. For example, consider the question

Which actress was married to Luc Besson while starring in one of his movies?

Researchers have recently developed several datasets of questions that require multiple relations to answer (e.g. Bao et al, 2016; Abujabal et al, 2017) and the use of time-based reasoning. Research is still at the early stages of these more complex questions datasets but some progress has been made (e.g. Zhang et al, 2017; Saxena et al, 2020).

7.3.3.2  Simple logical reasoning

Researchers have also studied how to perform simple types of reasoning on a KB in order to answer simple questions for which there is no fact stored in the KB. For example, suppose the question is

Is Lebron James an athlete?

and the KB does not contain this fact directly. Suppose further that the KB only has the fact

Lebron James is a basketball player

and in the knowledge base ontology the concept athlete has the sub-concept of basketball player. Then when the search finds the fact that Lebron James is a basketball player, it’s an easy inference to make use of this hierarchical information and conclude that Lebron James is also an athlete. Of course, this type of inference only works for hierarchical relationships.

CMU researchers (Sun et al, 2018) explored supplementing knowledge bases with entity-linked text. Their approach was to extract entities and relations from this text and use them to enhance the knowledge base prior to answering the question from the knowledge base.

He et al (2021) explored using a pair of networks:  A student network that finds the correct answer and a teacher network that learns reasoning rules.

For more in-depth overviews of complex KBQA, see Lan et al (2021) and Fu et al (2020).

7.4 Text question answering

As discussed above, KB’s are both limited in scope and limited in completeness and are hard to keep up-to-date. There is far more information available in text data sources such as Wikipedia. While structured KB’s are easier to query than text documents, by nature they provide far less coverage than text documents.

7.4.1 Closed domain question answering

Early research into answering queries from text documents focused on building systems that could perform reading comprehension tests. Reading comprehension research is focused on building systems capable of answering questions in reading comprehension settings like those commonly found in school settings. A block of text is presented. A question is asked about the text. The answer can be found in the text and no external knowledge is required. Perhaps the most prominent reading comprehension dataset is the Stanford Question Answering Dataset (SQuAD) (Rajpurkar et al, 2016). SQuAD is large enough to support machine learning with 107,000 crowdsourced questions based on passages from 536 Wikipedia articles. An example is shown below:

Passage: In meteorology, precipitation is any product of the condensation of atmospheric water vapor that falls under gravity. The main forms of precipitation include drizzle, rain, sleet, snow, graupel and hail… Precipitation forms as smaller droplets coalesce via collision with other rain drops or ice crystals within a cloud. Short, intense periods of rain in scattered locations are called “showers”.

Question: What causes precipitation to fall? (Answer: gravity)

Question: Where do water droplets collide with ice crystals to form precipitation? (Answer: “within a cloud”)

Figure 7.1. A SQuAD example.

Since the answer can always be found in the text, it is considered a closed-domain dataset.

A later version (SQuAD 2.0) added 50,000 questions that could not be answered from the passage for which the correct response was “not answerable”.

The development of large datasets like SQuAD and CNN/Daily Mail has enabled the application of neural networks to these tasks.

Most reading comprehension systems use the encoder-decoder architecture that was originally developed for machine translation. The encoder learns word embeddings of both the question and the text. The decoder uses a hard-coded or learned attention mechanism to find the portion of text that is most like the query (Hermann et al, 2015; D. Chen et al, 2016; Kadlec et al, 2016; Xiong et al, 2017; Seo et al, 2017; W. Wang et al, 2018). The limitation of the end-to-end learning approach is that these systems cannot generalize outside their training domains.

One of the main contributions of research on SQuAD and similar datasets that require finding the answer in a body of text is the development of systems that can zero in on the exact text string that answers the question.

AI systems now perform at a level that is actually better than human-level performance on the SQuAD 2.0 dataset. At first blush, it appears that AI systems that can perform at human levels are really reading and understanding these passages.

However, it turns out this is not really the case. When the answer is contained in a span of words in the document, human-level understanding is not required. Systems can become quite proficient just by learning how to locate the sentence and to match and align the lexical units in the sentence to those in the question.

This can be done by several techniques including direct lexical matching, use of syntactic cues, and use of word similarities based on the word embeddings associated with each word in the passage, question, and alternate answers.

If one looks closely at the example in Figure 7.1, one can see that answers can be obtained by surface-level analysis rather than true understanding. The correct sentence can be identified by matching the words “precipitation” and “fall” for the first question and the words “water”, “droplets”, “collide” (“collision”), “ice”, “crystals”, and “precipitation,” in the passage.

A similar dataset was created by Google DeepMind researchers by extracting news articles from CNN and The Daily Mail (Hermann et al, 2015). Both publishers provide bullet-point summaries of their articles. The researchers simply deleted an entity reference from a summary to create a question. For example:

Passage: The BBC producer allegedly struck by Jeremy Clarkson will not press charges against the “Top Gear” host, his lawyer said Friday. Clarkson, who hosted one of the most-watched television shows in the world, was dropped by the BBC Wednesday after an internal investigation by the British broadcaster found he had subjected producer Oisin Tymon “to an unprovoked physical and verbal attack.” . . .

Question: Producer X will not press charges against Jeremy Clarkson, his lawyer says.

Answer: Oisin Tymon

Researchers studying this dataset concluded that most questions systems could answer most questions by finding the single most relevant sentence in the passage.

This greatly restricts the number of possible entities in the answer – often to just one entity (D. Chen et al, 2016). Therefore, the task doesn’t really require understanding the passage. It merely requires finding the sentence that most closely matches the question. Finding the sentence can again be accomplished via word matching, syntactic cues, and/or similarities based on word embeddings.

A different Stanford Research effort (D. Chen et al, 2016) studied a random 100-question sample of the CNN/Daily Mail dataset and concluded that most answerable questions (25% were deemed unanswerable even by people) could be answered by finding the single most relevant sentence in the passage thereby greatly restricting the number of possible entities in the answer – often to just one entity.

Therefore, the task doesn’t really require understanding the passage. It merely requires finding the sentence that most closely matches the question.

Finding the sentence can be accomplished via lexical matching, syntactic cues, and/or similarities based on word embeddings.

Another set of Stanford University researchers (Jia and Liang, 2017) did some experiments to determine whether the systems scoring highest on the SQUAD dataset were really engaging in human-like natural language understanding. They added a syntactically correct but irrelevant and factually incorrect sentence to each passage. For example, here is an original SQUAD passage:

Passage: Peyton Manning became the first quarterback ever to lead two different teams to multiple Super Bowls. He is also the oldest quarterback ever to play in a Super Bowl at age 39. The past record was held by John Elway, who led the Broncos to victory in Super Bowl XXXIII at age 38 and is currently Denver’s Executive Vice President of Football Operations and General Manager.

Question: What is the name of the quarterback who was 38 in Super Bowl XXXIII?

Answer: John Elway

The Stanford researchers added this sentence to the passage:

Quarterback Jeff Dean had jersey number 37 in Champ Bowl XXXIV

(Jeff Dean wasn’t a quarterback. He was actually the head of Google AI research.) They found that this added sentence caused performance to decrease by over 50% and led the system to give answers like “Jeff Dean” to the question. They concluded that these systems are just learning very superficial rules (e.g. taking the last entity mentioned) and not exhibiting deep understanding and/or reasoning at all.

See Liu et al (2021) for a review of reading comprehension research.

7.4.2 Open-domain question answering

Open domain question answering (ODQA) involves answering questions using Wikipedia or some other unstructured text source. Answering questions by searching Wikipedia or the entire web can provide answers to a much wider range of questions than answering questions using KB’s (see above). Additionally, the information is often more up-to-date (though conflicting information can be a problem).

ODQA researchers often use queries from the KBQA datasets such as WebQuestions from RC datasets like SQuAD and Natural Questions (Kwiatkowski et al, 2019). However, rather than answering the questions from a specific KB or from a provided RC passage, questions are answered by searching text sources such as Wikipedia and the internet.

7.4.2.1  Information retrieval + reading comprehension

Early research broke the ODQA task into having two components:

• Information retrieval: Identifying the document or passage most likely to contain the answer.
• Reading comprehension: Searching the document or passage for the text span that contains the answer.

For example, a group of Facebook researchers (D. Chen et al, 2017) created the DrQA system using Wikipedia as their unstructured text source. The questions were taken from the SQuAD dataset but no passages were provided. They also used only Wikipedia as a text source and did not rely on information redundancy like IBM’s Watson system which used info from many different sources.

The first step was for the Information Retrieval module was to search Wikipedia and find the five best articles. The system compared the bag of words vector that makes up the question to the bag of words vector that makes up each article to find the five best articles.

The second step was for the reading comprehension system (the document reader) to try and find the answer in each of the five articles using techniques like those discussed in the RC section above.  When the document reader is given the top 5 articles returned by the information retrieval module, accuracy was 27.1%.  That is far worse than the 70.0% performance achieved by the document reader on its own when applied to the closed domain SQuAD 1.1 dataset where a single paragraph is provided.

It was also tested on an open domain version of the WebQuestions dataset where it produced 19.5% accuracy and on the WikiMovies dataset (Miller et al, 2016) where it achieved 34.3%. These low accuracy results are a good indication of how much harder it is to retrieve answers by searching all of Wikipedia than searching a single paragraph.

A group of Facebook researchers (Raison et al, 2018) developed the Weaver system as a better document reader. Their system uses very similar features to the DrQA document reader features. However, instead of encoding the question and context words separately, they use every combination of words composed of one word from the question and one from the context. These encodings are learned via multiple BiLSTMs stacked on top of each other plus a memory network.

Their system achieved an exact match accuracy of 74.4% of the original SQuAD dataset (paragraphs provided) compared to 70.0% for DrQA. It achieved 42.3% against the full Wikipedia vs. 27.1% for DrQA. It also achieved 23.7% on open domain WebQuestions (vs 20.7% for DrQA) and 43.9% on WikiMovies (vs 34.3% for DrQA).

A group of IBM researchers (S. Wang et al, 2018) looked to improve on the accuracy of these systems by using evidence-based re-ranking techniques reminiscent of (but quite different than) those used by the IBM Watson system. They were able to achieve 42.3% exact match accuracy on Quasar-T (compared to 51.5% human performance), 57.0% on SearchQA (compared to 43.9% human performance), and 44.6% on TriviaQA (human performance not measured).

7.4.2.2   Pre-trained language models + external knowledge sources

More recently, researchers have achieved state-of-the-art resulting using using pre-trained language models augmented by external knowledge sources.  Building on the success of earlier systems for retrieval augmented generation (Guu et al, 2020; Piktus et al, 2021; Borgeaud et al, 2022), Meta AI researchers (Izacard et al, 2022) created Atlas.  This system used an external knowledge source, Wikipedia, to help it answer questions and check facts:

Atlas has two components, a pre-trained language model and a retriever module.  The language model was trained using a T5 (Raffel et al, 2020) encoder-decoder text-to-text network.  It was pre-trained on Wikipedia articles using a self-supervised algorithm that used masked language modeling.  For each span of text in each document, three words were masked, and the system was trained to generate the three words.

The retriever module was trained with a dual-encoder architecture with one encoder for the question and one encoder for the documents.  It was trained simultaneously with the language model and the language model provided the supervision during the training.  More specifically, during the language model training, the question is concatenated with each document.  If the language model finds a document to be helpful in recovering the masked tokens, then the retriever’s objective function will rank the document higher and use that objective function to create similar encodings for queries and highly ranked documents.

When performing the question answering task, the model first retrieves the most relevant documents based on the similarity between the encoding of the question and each document.  Then the documents plus the question are fed to the language model which generates the output.

With only 64 fine-tuning examples, the Atlas model achieved state of the art results as of August, 2022 for few-shot question answering on several few-shot benchmarks.  On the NaturalQuestions benchmark, it achieved an accuracy of 42.4% and outperformed the PaLM language model (Chowdhery et al, 2022) by 3 percentage points despite having only 11 billion parameters compared to the 540 billion parameters in the PaLM model.

Using the full dataset for fine-tuning, Atlas achieved an accuracy of 64.0% which was 8.1 points better that the previous state of the art.

7.5 Question answering summary

This chapter discussed the inner workings of several question answering systems. First up was the IBM Watson system that beat the Jeopardy! champions. This was an amazing accomplishment. The IBM Watson system answered questions using primarily lexically-oriented techniques such as keyword matching. It could perform temporal reasoning using a set of handcrafted rules for comparing time periods or intervals.

This chapter also discussed the question answering systems developed out of research efforts that followed the IBM Watson system. These question answering techniques have greatly expanded the state of the art.

Knowledge base question answering techniques have improved to the point where entities and relationships in questions can be reliably extracted and matched to knowledge bases.

Reading comprehension techniques enable pinpointing the answer to a short span of text rather than just retrieving relevant documents.

In the decade following IBM’s Jeopardy! win, amazing progress has been made.  Rather than relying on handcrafted rules, today’s question answering systems now employ deep learning models that use external knowledge sources.

Chatbots >

© aiperspectives.com, 2020. Unauthorized use and/or duplication of this material without express and written permission from this site’s owner is strictly prohibited. Excerpts and links may be used, provided that full and clear credit is given with appropriate and specific direction to the original content.

SHARE THIS