Use next in a sentence for each word

Used with nouns:

«The next thing I knew, I was waking up.«
(thing)

«In my next life, I hope I am a bird.«
(life)

«We played the next round of golf.«
(round)

«She left the next day.«
(day, morning)

«The next generation will thank us.«
(generation)

«What is the next step?«
(step, move)

«Take the next turn.«
(turn, right, left)

«She is in the next room.«
(room)

«I have plans next weekend.«
(weekend, week, month)

«I haven’t seen the next episode.«
(episode, one)

«I will see you next time.«
(time)

«We met the next evening.«
(evening, night, morning)

«Get off at the next station.«
(station, stop)

«I read the next page.«
(page, paragraph)

«We will start again next year.«
(year, semester, quarter)

1   ‘We’ll be travelling round Europe next month,’ said Jerry.   following

      Jerry said that ……………………………………………… be travelling round Europe.

2   ‘I had Evan and Christie over for dinner last night,’ said Liz.   before

      Liz said that ……………………………………………… had Evan and Christie over for dinner.

3   ‘Daz came here two days ago and then suddenly left,’ said Barry.   gone

      Barry said that Daz had ……………………………………………… and then suddenly left.

4   ‘We’re going on our yearly diet tomorrow,’ said Jessie and Sandy together.   starting

      Jessie and Sandy said together that ……………………………………………… yearly diet the next day.

5   ‘I can pick you two boys up from school this afternoon,’ said their father to George and Kevin.   that

      George and Kevin’s father told his sons ……………………………………………… up from school that afternoon.

6   ‘I’ll buy these as they’re so cheap!’ said Toby.   going

      Toby said he ……………………………………………… as they were so cheap.

7   ‘You must study harder, Dave,’ said Dave’s mum.   him

      Dave’s mum ……………………………………………… study harder.

8   ‘I think you may be coming down with flu,’ Greg said to me.   thought

      Greg said that ……………………………………………… be coming down with flu.

9   ‘I don’t know why they haven’t contacted me recently,’ said Tine.   been

      Tine said she didn’t know why ……………………………………………… recently.

10   ‘Everything was different yesterday,’ said Ben.   been

        Ben said everything ……………………………………………… before.

You can find all the code at the end of the answer.

Most of your questions (why a Softmax, how to use pretrained embedding layer, etc…) were answered I reckon. However as you were still waiting for a concise code to produce generated text from a seed, here I try to report how I ended up doing it myself.

I struggled, starting from the official Tensorflow tutorial, to get to the point were I could easily generate words from a produced model. Fortunately after taking some bits of answer in practically all the answers you mentioned in your question, I got a better view of the problem (and solutions). This might contains errors, but at least it runs and generates some text…

how do I use the produced model to actually generate a next word suggestion, given the first few words of a sentence?

I will wrap the next word suggestion in a loop, to generate a whole sentence, but you will easily reduce that to one word only.

Let’s say you followed the current tutorial given by tensorflow (v1.4 at time of writing) here, which will save a model after training it.

Then what is left for us to do is to load it from disk, and to write a function which take this model and some seed input and returns generated text.

Generate text from saved model

I assume we write all this code in a new python script. Whole script at the bottom as a recap, here I explain the main steps.

First necessary steps

FLAGS = tf.flags.FLAGS
FLAGS.model = "medium" # or whatever size you used

Now, quite importantly, we create dictionnaries to map ids to words and vice-versa (so we don’t have to read a list of integers…).

word_to_id = reader._build_vocab('../data/ptb.train.txt') # here we load the word -> id dictionnary ()
id_to_word = dict(zip(word_to_id.values(), word_to_id.keys())) # and transform it into id -> word dictionnary
_, _, test_data, _ = reader.ptb_raw_data('../data')

Then we load the configuration class, also setting num_steps and batch_size to 1, as we want to sample 1 word at a time while the LSTM will process also 1 word at a time. Also creating the input instance on the fly:

eval_config = get_config()
eval_config.num_steps = 1
eval_config.batch_size = 1
model_input = PTBInput(eval_config, test_data)

Building graph

To load the saved model (as saved by the Supervisor.saver module in the tutorial), we need first to rebuild the graph (easy with the PTBModel class) which must use the same configuration as when trained:

sess = tf.Session()
initializer = tf.random_uniform_initializer(-eval_config.init_scale, eval_config.init_scale)
# not sure but seems to need the same name for variable scope as when saved ....!!
with tf.variable_scope("Model", reuse=None, initializer=initializer):
    tf.global_variables_initializer()
    mtest = PTBModel(is_training=False, config=eval_config, input=model_input)

Restoring saved weights:

sess.run(tf.global_variables_initializer())
saver = tf.train.Saver()
saver.restore(sess, tf.train.latest_checkpoint('../Whatever_folder_you_saved_in')) # the path must point to the hierarchy where your 'checkpoint' file is

… Sampling words from a given seed:

First we need the model to contain an access to the logits outputs, or more precisely the probability distribution over the whole vocabulary.
So in the ptb_lstm.py file add the line:

# the line goes somewhere below the reshaping "logits = tf.reshape(logits, [self.batch_size, ..."
self.probas = tf.nn.softmax(logits, name="probas")

Then we can design some sampling function (you’re free to use whatever you like here, best approach is sampling with a temperature that tends to flatten or sharpen the distributions), here is a basic random sampling method:

def sample_from_pmf(probas):
    t = np.cumsum(probas)
    s = np.sum(probas)
    return int(np.searchsorted(t, np.random.rand(1) * s))

And finally a function that takes a seed, your model, the dictionary that maps word to ids, and vice versa, as inputs and outputs the generated string of texts:

def generate_text(session, model, word_to_index, index_to_word, 
                  seed='</s>', n_sentences=10):
    sentence_cnt = 0
    input_seeds_id = [word_to_index[w] for w in seed.split()]
    state = session.run(model.initial_state)

    # Initiate network with seeds up to the before last word:
    for x in input_seeds_id[:-1]:
        feed_dict = {model.initial_state: state,
                     model.input.input_data: [[x]]}
        state = session.run([model.final_state], feed_dict)

    text = seed
    # Generate a new sample from previous, starting at last word in seed
    input_id = [[input_seeds_id[-1]]]
    while sentence_cnt < n_sentences:
        feed_dict = {model.input.input_data: input_id,
                     model.initial_state: state}
        probas, state = session.run([model.probas, model.final_state],
                                 feed_dict=feed_dict)
        sampled_word = sample_from_pmf(probas[0])
        if sampled_word == word_to_index['</s>']:
            text += '.n'
            sentence_cnt += 1
        else:
            text += ' ' + index_to_word[sampled_word]
        input_wordid = [[sampled_word]]

    return text

TL;DR

Do not forget to add the line:

self.probas = tf.nn.softmax(logits, name='probas')

In the ptb_lstm.py file, in the __init__ definition of PTBModel class, anywhere after the line logits = tf.reshape(logits, [self.batch_size, self.num_steps, vocab_size]).

The whole script, just run it from the same directory where you have reader.py, ptb_lstm.py:

import reader
import numpy as np
import tensorflow as tf
from ptb_lstm import PTBModel, get_config, PTBInput

FLAGS = tf.flags.FLAGS
FLAGS.model = "medium"

def sample_from_pmf(probas):
    t = np.cumsum(probas)
    s = np.sum(probas)
    return int(np.searchsorted(t, np.random.rand(1) * s))

def generate_text(session, model, word_to_index, index_to_word, 
                  seed='</s>', n_sentences=10):
    sentence_cnt = 0
    input_seeds_id = [word_to_index[w] for w in seed.split()]
    state = session.run(model.initial_state)

    # Initiate network with seeds up to the before last word:
    for x in input_seeds_id[:-1]:
        feed_dict = {model.initial_state: state,
                     model.input.input_data: [[x]]}
        state = session.run([model.final_state], feed_dict)

    text = seed
    # Generate a new sample from previous, starting at last word in seed
    input_id = [[input_seeds_id[-1]]]
    while sentence_cnt < n_sentences:
        feed_dict = {model.input.input_data: input_id,
                     model.initial_state: state}
        probas, state = sess.run([model.probas, model.final_state],
                                 feed_dict=feed_dict)
        sampled_word = sample_from_pmf(probas[0])
        if sampled_word == word_to_index['</s>']:
            text += '.n'
            sentence_cnt += 1
        else:
            text += ' ' + index_to_word[sampled_word]
        input_wordid = [[sampled_word]]

    print(text)

if __name__ == '__main__':

    word_to_id = reader._build_vocab('../data/ptb.train.txt') # here we load the word -> id dictionnary ()
    id_to_word = dict(zip(word_to_id.values(), word_to_id.keys())) # and transform it into id -> word dictionnary
    _, _, test_data, _ = reader.ptb_raw_data('../data')

    eval_config = get_config()
    eval_config.batch_size = 1
    eval_config.num_steps = 1
    model_input = PTBInput(eval_config, test_data, name=None)

    sess = tf.Session()
    initializer = tf.random_uniform_initializer(-eval_config.init_scale,
                                            eval_config.init_scale)
    with tf.variable_scope("Model", reuse=None, initializer=initializer):
        tf.global_variables_initializer()
        mtest = PTBModel(is_training=False, config=eval_config, 
                         input_=model_input)

    sess.run(tf.global_variables_initializer())

    saver = tf.train.Saver()
    saver.restore(sess, tf.train.latest_checkpoint('../models'))

    while True:
        print(generate_text(sess, mtest, word_to_id, id_to_word, seed="this sentence is"))
        try:
            raw_input('press Enter to continue ...n')
        except KeyboardInterrupt:
            print('bbQuiting now...')
            break

Update

As for restoring old checkpoints (for me the model saved 6 months ago, not sure about exact TF version used then) with recent tensorflow (1.6 at least), it might raise an error about some variables not being found (see comment).
In that case, you should update your checkpoints using this script.

Also, note that for me, I had to modify this even further, as I noticed the saver.restore function was trying to read lstm_cell variables although my variables were transformed into basic_lstm_cell which led also to NotFound Error. So an easy fix, just a small change in the checkpoint_convert.py script, line 72-73, is to remove basic_ in the new names.

A convenient way to check the name of the variables contained in your checkpoints is (CKPT_FILE is the suffix that comes before .index, .data0000-1000, etc..):

reader = tf.train.NewCheckpointReader(CKPT_FILE)
reader.get_variable_to_shape_map()

This way you can verify that you have indeed the correct names (or the bad ones in the old checkpoints versions).