Intrinsic motivation ( , ) has emerged as one explanation for humans’ and animals’ impressive learning capabilities. Steered by the need to explore their environment (whether this need is satiable or not has been a fierce debate in the behavioral psychological community ( , )), they are able to discover near-optimal strategies in very efficient ways. Designing artificial agents presenting such capabilities is a central goal of modern artificial intelligence and Reinforcement Learning (RL) is a popular candidate to do so. RL has addressed a variety of sequential-decision-making problems whether in games ( , ) or robotics ( , ). Nevertheless, some simple problems remain unsolved. Current state-of-the-art methods struggle to find good policies in environments (1) where constant negative rewards may discourage the agent to explore (e.g., the Pitfall! game from Atari), (2) where the reward is so sparse that an agent does not find any (e.g., the Montezuma’s Revenge Atari game), (3) where state and action space are big (e.g., text worlds). These tasks remain fairly easy for humans, though. In order to tackle these specific problems, the use of reward bonuses, inspired by animal curiosity, was proposed to steer the agent’s exploration  ( , ). Even though different intrinsic bonuses have been proposed, a large majority rely on the same principle: reward for novelty. These methods mostly differ in how they compute this notion of newness. Count-based methods do it by counting how often the agent has encountered a given state  ( , ). Pseudo-counts methods  ( , ) allow to approximate counts in large state spaces. Prediction error is also used to measure novelty, either by computing the agent’s ability to predict the future ( , ) or random statistics about the current state ( , ). Some restrict novelty to state-action pairs that have an impact on the agent ( , ) or derive empowerment metrics ( , ) using mutual information. All these methods naturally encourage the discovery of new states through exhaustive exploration. Yet, in most realistic environments, exhaustive exploration is (1) not feasible due to the size of the state-action space, (2) not desirable as most behaviors are unlikely to be relevant for the task at hand.

Nonetheless, human and more generally mammals exploration behaviors are governed by various motivations and constraints. Intelligent Beings do not have unlimited resources of time and energy. They optimize these resources to survive and reproduce but also to have fun ( , ), to help others ( , ) or to satisfy their curiosity. ( ) make the difference between homeostatic motivations (that encourage to stay in the “comfort zone” and generally correspond to desires that can be satiated) and heterostatic motivations (that push organisms out of equilibrium but cannot be satiated). These many desires shape the way organisms interact with their environment, encouraging them to discover new things but also to protect themselves, avoiding over-surprising events with mechanisms like fear ( , ). ( ) exemplified how to exploit such priors by implementing a “homeostasis” objective for RL, thereby showing how different from “novelty seeking” these priors can be. Eventually, the resource constraints stop organisms from exploring exhaustively their environment and push them to transfer knowledge from past experience. ( ) developed the idea of optimal incongruity: high-novelty is not rewarded as much as intermediate-level novelty, suggesting how curiosity is tightly connected to fear. More recently, ( ) supported this hypothesis with experiments on children curiosity. Overall, novelty methods fail to model correctly human curiosity as they consider that “the newer, the better”. This failure calls for a new way of definingour agent’s intrinsic motivations.

In an arbitrary environment, exhaustive exploration is desirable and leads to convergence with theoretical guarantees ( , ). But when the exploration presents some structure, one can transfer skills and priors from similar environments. ( ) exemplified, in the case of simple video games, how humans priors help us to solve new problems. The authors enlighten how humans struggle to play the same underlying video game with change of the object semantics, physics modifications (e.g., the gravity is rotated) or with visual similarities transformations. Overall, they show how much of the human’s ability to solve a new game in a zero-shot manner is due to their prior on the environment.

In this paper, instead of hard coding what we think an agent’s motivations should be (e.g. novelty), we propose to learn a bonus that captures these sources of motivation from demonstrations. By adopting this approach, we expect to learn a bonus that implicitly helps reproducing a structured exploration behavior (i.e. using priors from the demonstrations to reduce the search space), in lieu of an exhaustive one. We also argue that, to a certain extent at least, this can happen without the need of extra modelling inspired by cognitive or behavioral research. To do so, we cast this problem as an inverse RL problem with the difference that only some fraction of the reward optimized by an observed agent is hidden: the intrinsic motivation bonus. The task-related reward remains provided by the environment. We then build upon ( ) to propose a method that allows us to recover the intrinsic motivation from demonstrations.

Therefore, our contributions are the following:

To evaluate SmtW, we validate a set of hypotheses on a controlled environment. We notably find that our method can learn structures and styles, transfer useful priors and encourages long-term planning.

Markov Decision Processes. In Reinforcement Learning (RL), an agent learns to behave optimally through interactions with an environment. This is usually formalized as a Markov Decision Process (MDP) ( , ), a tuple $(S,A,P,R,\gamma )$ with $S$ the set of states, $A$ the set of actions (assumed discrete here), $:P\to \times SAP(S)$ the Markovian transition kernel defining the dynamic of the environment, $:R\to \times SAR$ a bounded reward function and $\in \gamma [0,1[$ a discount factor. The agent interacts with the environment through a (here deterministic) policy $:\pi \to SA$. The quality of a given policy is quantified by the associated state-action value function, or $Q$-function. It is the expected discounted cumulative reward for starting from $s$, taking action $a$, and following $\pi$ afterward: $Q\pi (s,a)=E\pi [\sum \ge t0\gamma trt|s0=s,a0=a]$, with $=at\pi (st)$, $=rtR(st,at)$ and $s+t1\sim P(.|st,at)$. By construction, it satisfies the Bellman equation: for any $s,a$, $=Q\pi (s,a)+R(s,a)\gamma \sum s\prime P(|s\prime s,a)Q\pi (s\prime ,\pi (s\prime ))$. An optimal policy $\pi *$ satisfies component-wise $\ge Q\pi *Q\pi$, for any policy $\pi$. Let $=Q*Q\pi *$ be the associated (unique) optimal $Q$-function, any deterministic optimal policy is greedy with respect to it: $\in \pi *(s)argmaxaQ*(s,a)$.

Exploration Bonus. A common strategy to encourage exploration is to augment the reward function with a bonus. This bonus generally depends on past history. For example, a bonus rewarding novelty requires remembering what has been experienced so far. Write $=ht(s0,a0,\dots ,s-t1,a-t1,st)$ the history up to time $t$, and $H$ the set of all histories. Generally speaking, we abstract a bonus as $:B\to \times HAR$, and use it for addressing the dilemma between exploration and exploitation, which thus amounts for the agent to optimize for $+R(st,at)B(ht,at)$ instead of simply $R(st,at)$. This state-of-the-art exploration bonuses all rely on memory, e.g. by counting the state visitation  ( , ) or through updates of a neural networks  ( , ). These exploration bonuses are designed to express the prior that any source of novelty is good for exploration.

Such a prior on what is good for exploration is task-specific. ( ) showed that state-of-the-art bonuses were degrading performances in most Atari games.

Rather than handcrafting a bonus that encodes what we think intrinsic motivation should be (e.g. using novelty), we propose to learn it from demonstrations of exploratory behaviours.

We thus assume that the demonstrator learns to solve a task by exploring its environment and a simple solution would be to perform behavior cloning. Because the demonstrator is likely to use past interactions to make decisions (remembering what has been already tried so far), we could frame our problem as learning, in a supervised manner, a mapping from histories to actions. Yet, behavioral cloning suffers the behavioral drift ( , ), which would be exacerbated in the case of history dependent policies . Moreover, we would like to transfer this behavior to new environments and possibly to new tasks.

Imitation learning classically assumes that experts are optimizing an MDP with an unknown reward function $RE(s,a)$. Note that this introduces a modelling bias, i.e. a human performing a task is not necessarily explicitly solving an MDP. In this study, we do not tackle the standard problem of inferring an optimal behavior from demonstrations, but of estimating an exploratory behavior. Yet, the latter can be reduced to the former. Taking inspiration from RL, and especially from the body of work about the exploration-exploitation dilemma, we assume that our demonstrator is optimizing for an unknown reward function $RE(s,a)$, standing for the task objective, augmented with a trajectory-dependent intrinsic bonus $BE(h,a)$, standing for how the environment is explored. Making this bonus depend on past interactions is important as we can reasonably assume that exploration is based on memory (one would not try to always reproduce situations that were already seen). Then, we assume that the expert is optimal for the bonus-augmented reward $+RE(s,a)BE(h,a)$, in the augmented MDP $\{H,A,P,+REBE,\gamma \}$. That is the original MDP, with the state space being replaced by the set of all trajectories and the reward function being augmented with a bonus function. So, we reduced our original problem to Inverse Reinforcement Learning (IRL): learn a function $:^RB\to \times HAR$ such that the demonstrator is the (unique) optimal policy. By design, $=^RB+REBE$ is a solution to this problem (even if it is not learnable exactly, as the optimal policy is invariant to many reward transformations ( , )). Yet, we also assume that we know the task’s reward $R$, or at least that we observe it in the demonstrations. Formally, it may be different from the reward $RE$ (even if we assume that it leads to a similar optimal behavior), but we can leverage it to disentangle the task contribution and the exploration one. For doing so, we propose to learn a bonus function $:^B\to \times HAR$ such that the demonstrator is optimal for $+R^B$. Notice that it does not change the problem, as it is just a reparameterization of the previous one (by setting $=^B-^RBR$).

Thus, our IRL problem has additional constraints. First, we want to recover the bonus, for transferring to new environments or new tasks, this preclude using imitation learning methods that do not explicitly recover rewards (IRL is mandatory). Second, our specific parameterization ($+RB$) precludes using IRL methods that would not allow using the observation of the task reward $R$ along the expert trajectories. Third, the function we want to estimate is history-dependent which requires an IRL method able to use sequences as inputs.

Formalism. We assume to have access to demonstrations that are optimal according to the (known) reward of the environment plus an (unknown) intrinsic bonus. The environment being assumed Markovian, knowing the current state is enough to act optimally according to the task (optimizing for the environment’s reward). Yet, the demonstrator also optimizes its exploration bonus, that depends on the past. To formalize things, we consider that the demonstrations are provided by a policy $:\pi E\to HA$, and that the policy is optimal for the augmented MDP $(H,A,P,+REBE)$, where $H$ replaces $S$ and $+REBE$ replaces $R$.

We frame our problem as learning the bonus $BE$ from trajectories sampled from $\pi E$.

Our approach. If we cannot naively apply any existing IRL algorithm to our problem, it can be a source of inspiration. Especially, one suits well our problem: the set-policy framework ( , ). It shows that a formal bijection between supervised learning and IRL exists. Among the covered algorithms, the Cascaded Supervised approach to IRL (CSI) ( , ) is of particular interest to us. We refer the reader to these papers for more details and we explain in details here how the CSI paradigm can be readily applied to our setting.

The demonstrator’s policy, $\pi E$, is assumed optimal for $+REBE$ (which is unknown), so

Write $=QE(h,a)Q*+REBE(h,a)$ the associated optimal $Q$-function. It satisfies the Bellman optimality equation (writing $=h(\dots ,s)$, that is $s$ the last state of the trajectory $h$ and $=h\prime (h,a,s\prime )$):

Would the optimal policy and $Q$-function be known, we could use them to recover the optimized bonus-augmented reward using this Bellman equation:

Now, the quantities in the right hand side are unknown, but they can be estimated in an indirect way.

Assuming that the actions are discrete, we can learn the policy $\pi E$ by mapping histories to actions (using, for example, an LSTM network). Write $:^\pi \to HS$ the resulting policy, or classifier. If we train it by minimizing a cross-entropy loss, what we learn indeed are logits $^Q(h,a)$, and the classifier is $=^\pi (h)argmaxa^Q(h,a)$. Said otherwise, $^\pi$ is greedy with respect to $^Q$, that can thus be interpreted as an optimal Q-function for an unknown reward. Using the Bellman equation, we can recover this reward:

By Bellman, as $^\pi$ is greedy w.r.t. $^Q$, we have that $^\pi$ and $^Q$ are respectively the optimal policy and $Q$-function for the reward $+R^B$. We cannot use directly Eq. (), the model being unknown, but we can sample the right hand side and estimate $^B$ by solving a regression problem.

Therefore, we have reduced our initial problem to a sequence of supervised learning problem. Our algorithm is indeed CSI, up to the fact that we consider trajectories instead of states, and parameterize the bonus with the reward task. As such, the theoretical results of ( ) applies to our setting. Notably, we would have that

with $ϵ1$ the classification error and $ϵ2$ the regression error. This means that the demonstrator policy is close to optimal if these errors are small, for the learnt bonus function. One could argue that this bound trivially holds for $=+R^B0$, when all behaviors are optimal. Yet, this is unlikely, as for having the classification error $ϵ1$ small, we must have $^Q(h,\pi E(h))>^Q(h,a\ne \pi E(h))$ w.h.p., and thus learn an informative bonus.

Implementation. More concretely, we consider $softmax(Q\varphi )$ to be a neural network classifier with LSTM ( , ) units, $\varphi$ being the set of parameters and $Q\varphi$ being the logits. We train $\pi \varphi$ to do behavioral cloning, that is to predict the demonstrator actions $aE$ based on its past interactions $hE$, by minimizing a cross-entropy loss:

with $Q\varphi (h,a)$ the $ath$ logit for input $h$. If the classifier learns correctly, the logits of the resulting network should satisfy $>Q\varphi (hE,aE)Q\varphi (hE,a)$ for $\ne aaE$, and the class predicted by the classifier will be $=\pi \varphi (h)argmaxaQ\varphi (h,a)$. Hence, as explained above, one can interpret $Q\varphi$ as an optimal $Q$-function (hence the notation), and $\pi \varphi$ as the associated optimal policy. Recall that these quantities can be related to the bonus-augmented reward through Bellman:

Then, we can learn a network $B\theta$ (parameterized by $\theta$, with LSTM units) by minimizing a square-loss, the regression target being $-Q\varphi (hE,aE)\gamma Q\varphi (hE\prime ,\pi \varphi (hE\prime ))R(sE,aE)$, an unbiased sample of what would give the true Bellman equation. However, we only observe optimal actions (according to $+RB$), so this alone would hardly generalize to suboptimal ones. Therefore, we propose a heuristic, that consists in regressing for suboptimal actions towards $Bmin$, a hyperparameter of the algorithm. For example, it could be set to $min(-Q\varphi (hE,aE)\gamma Q\varphi (hE\prime ,\pi \varphi (hE\prime ))R(sE,aE))$, the minimum being over transitions in the dataset. This gives the following loss, for a transition $(hE,aE,hE\prime )$, and for $¯aE$ being sampled randomly in $\setminus A\{aE\}$:

To sum up, we train a BC policy by minimizing $LBC$. The implicit resulting logits are considered optimal $Q$-values, that are in turn used to learn the bonus $B\theta$ by minimizing the loss $Lreg$ (Figure ).

We aim at providing insights on what priors SmtW is able to extract from the demonstrations and specifically, we wish to verify that SmtW is able to encourage a structured exploration of the environment. In order to thoroughly study the method, we test it on a grid-world where we are able to design controllers with specific behaviors. As in IRL, studying the return of an agent trained with our bonus is only a proxy to evaluate SmtW’s quality and is not informative on the priors the bonus conveys. We thus focus our experiments on analyzing the priors that were extracted from the demonstrations by the method. More specifically we wish to answer the following questions: (1) Is SmtW encouraging the demonstrator’s behavior more than a random one? (2) Is SmtW capturing the demonstrator’s style, its way of exploring the environment? (3) Does SmtW capture the skills required to solve the task? (4) Does SmtW encourage novelty seeking? (5) Does SmtW capture the constraints the demonstrator may be submitted to? To do so, we design controlled behaviors and study the bonus returned along these specific behaviors by SmtW, as described in Fig. . Given a behavior $A$ and a behavior $B$, this allows to check if a given bonus encourages behavior $A$ over $B$ or vice versa or rewards them equivalently.

After addressing these questions, we also verify that a simple agent can benefit from SmtW to actually solve efficiently a task.

The environment. We introduce a specific environment to answer these. We require this environment to be procedurally-generated in order to test SmtW’s ability to generalize to unseen environments. We require the environment to be complex enough so that exhaustive exploration is prohibitively expensive. To achieve this, we introduce the KeysDoors grid-world of size NxN.

It contains $N$ keys and $N$ doors, modeled by two different colors. The agent has a third color. The goal is to find the correct key and to open the correct door with it. As doors (resp. keys) are indistinguishable (except by their locations), an explorer has to try the different keys on the different doors. Actions available are {go left, go right, go up, go down, take, open, wait}. When an agent makes the action “take” on a key, it is then able to move with it. Actions “open” or “take” make the agent lose the key it was previously holding. To solve the task, the agent has to go to the correct key, take it, go to the correct door without doing action “take” or “open” on the way (so as not to lose the key), and then “open” the door. We need the environment to require perseverance so we made the reward function -1 for any actions but the wait action, that is rewarded 0. Opening the correct door with the correct key gives a reward of 100 and terminates the episode. It requires perseverance as a “lazy” policy would get a return of 0 whereas trying to find the 100 reward gives -1 at each step. This is a well known issue in RL that simple exploration leads to such lazy solutions.

The KeysDoors environment is generated procedurally. For each column, locations for a door and a key are sampled uniformly without replacement. Thus, there is exactly one key and one door on each column and these cannot be at the same location. The “correct” key is then uniformly sampled among the keys and the "correct" door is sampled uniformly among the doors. The initial position of the agent is sample uniformly on the grid. The environment gives both a ground-truth-state (an integer representing the current state), unused by SmtW as well as an RGB observation (as shown in Fig. ), used by SmtW. Figure  shows a trajectory in one possible instance of the KeysDoors environment with $=N5$. Every observation $x$ (an $\times NN3$ tensor) is normalized between 0 and 1 by dividing by 255.

The demonstrations. For a given instance of the environment, the demonstrator navigates between keys and doors and tries key/door pairs in a precise order. It takes the first key on the left and tries it on the first door on the left, then it tries the same key on the second door etc. Once it has tried the first key on every door, it repeats the operation with the second key and proceeds further this way. The episode ends when the demonstrator finds the right key/door pair and obtains the reward. Then it “exploits”, taking the correct key and opening directly the correct door five consecutive times. Note that this also simulates the non-stationnarity happening in most goal-directed task solving processes. One first mainly explores and then exploits more and more.

Train vs. Test. The bonus is always used in new test environments, unseen in the demonstrations. SmtW’s ability to generalize to new environments is thus tested in all the following experiments. Given the possible positions of the keys, of the doors and then of the correct key and the correct door, there are $(-N1)N3$ possible instances of the environment. The behaviors that are designed to study what is actually encouraged or discouraged by SmtW are the following. Their associated bonus is always studied in test instances of the environment.

A trajectory of an agent moving to a key, taking it, moving to a door and trying to open it with the key is shown in Fig. 

Intrinsic Motivation. Intrinsic motivation is essential to mental development ( , ) and we can argue that this may, in consequence, be an essential component for computational learning. ( ) argue that all humans respond to intrinsic motivations. Young infants motivations can be qualified as more chaotic as they push children to bite, throw, grasp or shout in order to learn. Adults, in contrast, have more structured intrinsic motivations, activated, for instance, when they play games, read novels or watch movies. Correctly using these numerous intrinsic motivations can be key to train agents that solve more and more difficult tasks. Instead of modeling such intrinsic motivations to mimic cognitive processes, we learn them from demonstrations.

Exploration. In order to provide an exploration signal to the agent,  ( , ) proposed the very intuitive count-based method in order to measure novelty. Counting how many times the agent has been in a given state, it rewards less visited states. Several methods extended this idea to large state-space problems  ( , ), where it is not possible to count state occupancy. Intrinsic curiosity is also commonly computed as a prediction error, either trying to predict the environment’s dynamics  ( , ) or random statistics about the current state  ( , ). Different methods try also to measure surprise as a prediction gain  ( , ). Instead of designing such a bonus, we aim at learning one from demonstrations.

Learning from demonstrations. Imitation learning, the problem of learning from demonstrations, is typically folded into two different paradigms. (1) Behavioral cloning ( , ) tries to directly match the demonstrator’s behavior, generally using supervised learning techniques. (2) Inverse Reinforcement Learning  ( , ) first tries to recover a reward explaining the demonstrator’s behavior, before optimizing the reward for imitating the demonstrator. Some methods output an explicit reward  ( , ) while adversarial imitation learning can be seen as IRL with implicit reward recovery  ( , ). Overall these methods all assume that the near-optimality of the demonstrations. Some works try to relax this assumption and to learn from sub-optimal demonstrations  ( , ). IRL methods typically control the quality of their algorithm through the proxy of the return obtained by an agent trained on the inferred reward.

Our methods differs from these methods it does not assume that demonstrations are optimal but rather try to answer the question: “In what way is the demonstrator’s behavior deviating from an optimal policy?”. Moreover, we do not seek to recover a reward as in IRL but rather to recover a bonus explaining which, added to the environment reward, explains the demonstrator’s behavior. Facing the same problem that the usual proxy to control the algorithm quality (training an agent on the inferred bonus) is not informative, we decided to study our method through its response to various behaviors.

In this work, we present a novel method for extracting an intrinsic bonus from the demonstrations. The method we introduce is offline and does not require environment interactions to recover the bonus, unlike recent adversarial imitation methods who need numerous interaction in order to recover a reward function. Anyway, those methods could not be readily applied to our problem, as they do not explicitly compute a reward function. Moreover, to the best of our knowledge, this is one of the very first method to recover some kind of reward that is history-dependent. We show how this bonus generalizes to unseen environments and is able to convey long-term priors. We exemplified the approach on a simple yet didactic and challenging example. Yet, testing the method on a larger-scale environment would require human exploratory demonstrations. Gathering such a dataset is costly and very few are already available, none of them really covering our setting. Even though the given example is simple, this novel approach of capturing the demonstrator’s bias could potentially lead to new lines of work in RL. For instance, one could use our method to implement behavioral style-transfer in RL and show to an agent a specific way to solve the task thanks to demonstrations. Combining a reward and biases extracted from demonstrations may also help for robotic tasks, where some aspects of the task are easily programmable with a reward but some expectations on how to solve the task may be easier to transmit thanks to demonstrations. This could also lead to some advances in tackling mispecified rewards. Using both a reward, that would contain information on the task to solve but not fully describe the constraints of the problem and demonstrations to correct the reward can be key to train sequential controllers in complex dynamics.