Supplementary Material

1 section 1 1 §1 <tag close=" ">1</tag>Agent details 1.1 subsection 1.1 1.1 §1.1 <tag close=" ">1.1</tag>Fixed agent hyperparameters

See Table .

Table 1 Table 1 1 Table 1 1Agent hyperparameters that are fixed throughout our experimentation.

Table 1Agent hyperparameters that are fixed throughout our experimentation.
Hyperparameter	Value	Description
train_steps	640m	Theoretical maximum number of time steps (across all episodes) for which the agent will be trained.
env_steps_per_core_step	4	Number of time steps between each action decision (action smoothing)
num_workers	32	Number of independent workers running replicas of the environment with asynchronous updating.
unroll_length	50	Number of time steps through which error is backpropagated in the core LSTM action module
visual encoder
num_layers	3	Layers in the convolutional vision network.
output_channels	(32, 64, 64)	Number of feature maps in each layer of the network.
kernel_shapes	(8, 4, 3)	Shapes of the (square) convolutional kernels in each layer of the network.
strides	(4, 2, 1)	Convolution stride length in each layer of the network.
activation	relu	Activation function applied after all except the final layer of the visual encoder.
language encoder
embedding_dim	128	Dimension of the word and instruction embeddings.
cost calculation
additional_discounting	0.99	Discount used to compute the long-term return R_t in the A3C objective
cost_base	0.5	Multiplicative scaling of all computed gradients on the backward pass in the network
optimisation
clip_grad_norm	100	Limit on the norm of the gradient across all agent network parameters (if above, scale down)
decay	0.99	Decay term in RMSprop gradient averaging function
epsilon	0.1	Epsilon term in RMSprop gradient averaging function
learning_rate_finish	0	Learning rate at the end of training, based on which linear annealing of is applied
momentum	0	Momentum parameter in RMSprop gradient averaging function

1.2 subsection 1.2 1.2 §1.2 <tag close=" ">1.2</tag>Variable agent hyperparameters

Agent hyperparameters randomly sampled in order to yield different replicas of our agents for training. uniform( $x, y$ ) indicates that values are sampled uniformly from the range $[x, y]$ . loguniform( $x, y$ ) indicates that values are sampled from a uniform distribution in log-space (favouring lower values) on the range $[x, y]$ . Unless otherwise stated, 16 replicas were started for each experimental condition, and results reported reflect the best 5 of these replicas.

See Table .

Table 2 Table 2 2 Table 2 2Agent hyperparameters that randomly sampled in order to yield different replicas of our agents for training. uniform(

x, y

) indicates that values are sampled uniformly from the range

[x, y]

. loguniform(

x, y

) indicates that values are sampled from a uniform distribution in log-space (favouring lower values) on the range

[x, y]

Table 2Agent hyperparameters that randomly sampled in order to yield different replicas of our agents for training. uniform( $x, y$ ) indicates that values are sampled uniformly from the range $[x, y]$ . loguniform( $x, y$ ) indicates that values are sampled from a uniform distribution in log-space (favouring lower values) on the range $[x, y]$ .
Hyperparameter	Value	Description
language encoder
embed_init	uniform(0.5, 1)	Standard deviation of normal distribution (mean = 0) for sampling
		initial values of word-embedding weights in L
optimisation
entropy_cost	uniform(0.0005, 0.005)	Strength of the (additive) entropy regularisation term in the A3C cost function.
learning_rate_start	loguniform(0.0001, 0.002)	Learning rate at the beginning of training
		annealed linearly to reach learning_rate_finish at the end of train_steps.

Word class	Words
Shapes (40)	chair, suitcase, tv, ball, balloon, cow, zebra cake, can, cassette, chair, guitar, hair-brush,
	hat, ice-lolly, ladder, mug, pencil, suitcase, toothbrush, key, bottle, car, cherries, fork,
	fridge, hammer, knife, spoon, apple, banana flower, jug, pig, pincer, plant, saxophone,
	shoe, tennis-racket, tomato, tree, wine-glass
Colours (10)	blue, brown, pink, yellow, red, green, cyan, magenta, grey, purple
Patterns (2)	pinstriped, dotted
Shades (2)	lighter, darker
Directions (2)	left, right