How do babies learn to navigate varied terrain such as steep slopes, high drop-offs, wide gaps, and narrow bridges, as illustrated in Fig. ? Infants learn to walk amidst continual changes in their environments and skills. Features of the environment are variable (ground surfaces can be flat or sloping, rigid or deformable, high-traction or slippery; the path can be clear or cluttered with obstacles and elevations) and infants’ walking skill improves dramatically over the first several months after walk onset []. During natural locomotion, infants’ walking paths are curved, not straight; their steps are omnidirectional, not forward; and their locomotion is intermittent rather than continuous []. Infants’ everyday life creates a natural training regimen—with toys scattered on the ground, laundry heaps in the corner, furniture obstructing the path, and so on. For infants, learning to walk means learning to modify gait from step to step to navigate a varied environment.

Developmental researchers test novice and experienced walking infants on challenging terrain, such as slopes, drop-offs, gaps, and bridges of varying difficulties, because the obstacles are novel (no baby encounters steep slopes, narrow bridges, etc. during everyday life). Thus, researchers can test generalization from everyday experience to novel situations based on whether babies modify their gait prospectively while approaching the obstacle or reactively after stepping onto the obstacle. In all cases, the best solution is to modify gait prospectively on approach. Reactive adjustments while traversing the obstacles are less optimal because gravity takes over and gait modifications entail fighting to keep the body over the moving base of support. The strongest evidence for prospective control is changes in foot placement, step length, and walking speed before stepping onto the obstacle—rather than modifying gait after stepping onto the obstacle.

Prior work suggests the ability to modify gait develops with walking experience []. Novice infants walk blithely over the brink of impossibly steep slopes, high drop-offs, wide gaps, and narrow bridges (requiring rescue from an experimenter). After several months of everyday walking experience—notably, with no prior experience on the test obstacles—infants modify their gait prospectively. For every obstacle, experienced infants slow down and shorten their steps during approach. For slopes, their initial steps on the slope are short and slow, they widen their base of support, and brake forward momentum from step to step []. For drop-offs, they place their stance foot close to the brink so that their moving foot can stretch down to the bottom of the precipice all the while keeping their body upright as it drops vertically onto their moving foot []. For gaps, they place their stance foot close to the brink of the gap, increase step length with the moving foot, and place their moving foot close to the far side of the gap. For bridges, infants place their body at the near side of the bridge, and take short, slow, narrow steps [].

Developmental researchers hypothesize that infants learn to modify gait prospectively as they accumulate everyday experience navigating an ever-changing environment. Practice doing the same thing repeatedly (e.g., stepping on a treadmill) and training to the test (e.g., repeated practice walking down slopes) do not contribute to the development of prospective gait modifications []. However, developmental researchers cannot control infants’ natural, everyday training regimens and must accommodate practical limitations for testing infants. Babies can complete only a few dozen trials before becoming tired or fussy, and often learn gait modifications During testing. Such limitations do not exist for simulated robots. Thus, simulated robots provide a powerful, highly-controlled platform for testing effects of experience on motor learning, with the potential to serve as a surrogate for studying how infants learn to walk over varied terrain.

Controller Architecture We tested the behavior of controllers for the Cassie bipedal robot trained with RL using a learning framework from prior work [], lightly adapted for purely simulation-based experimentation. This framework layers a visually-guided residual controller on top of the outputs of a frozen blind controller trained only on flat terrain. The controllers are represented as recurrent long short-term memory (LSTM) neural networks. Their outputs are proportional derivative (PD) control targets, set at 50 Hz for 10 actuated joints centered around a standing pose. A static PD controller uses these targets to set the motor torques at 2000 Hz. A schematic of this learning framework is shown in Fig. . Training a slower controller to set targets for a faster PD controller is commonly used in RL for locomotion [] and is a strong precedent for the Cassie platform [].

The input to the blind controller includes: (1) a 35-dimensional vector containing the positions and velocities of all joints, plus the pelvis orientation and rotational velocity; (2) a clock signal that dictates the cadence of the footsteps; (3) gait parameters that modulate the clock signal; and (4) commands for forward speed, lateral speed, and turn rate. A grid of ground-truth terrain heights taken from the simulated terrain serves as additional input to the visually-guided controller. The grid is a 1m wide by 1.5m long rectangle of 20 by 30 values respectively in front of Cassie.

In addition to PD target residuals, the visually-guided controller outputs the clock progression speed and the phase-offset between the feet. This allows the controller to modulate the frequency of footsteps and adjust the left-right cadence. In principle, this means the controller can learn to produce asymmetric 2-beat patterns like skipping rather than performing only the basic, symmetrical, left-right gait pattern. Both blind and visually-guided controllers use the same reward function. It encourages adherence to the commands, minimized motor torque, and footsteps in accordance with the clock signal. See [] for further details.

Simulation Training The controllers are trained using the actor-critic proximal policy optimization (PPO) algorithm with gradient clipping, a standard model-free RL algorithm []. The simulator used for both training and testing is the MuJoCo physics engine [] using a model of the Cassie robot. Because we examined behavior only in simulation, we did not use dynamics randomization to aid in sim-to-real transfer as in prior work with Cassie [].

We trained Cassie on standard, multi-test obstacle, combined standard and multi-test, and single-test obstacle terrains. The standard training terrains replicate prior robotics work [] on which our learning framework is based: flat, hills, ridges, blocks, and stairs, shown in Fig. . The multi-test obstacle terrains are recreations of test apparatuses used in experiments with infants: slopes, drop-offs, gaps, and bridges, shown in Fig. for infants in the real world and Cassie in simulation.

Table shows the frequency of each terrain for the standard, multi-test obstacle, combined standard and multi-test, and single-test obstacle training regimens. Cassie received 20k iterations for the single-test obstacle regimen and 110k iterations for each of the others. The standard regimen uses a range of commands for forward speed, lateral speed, and turn rate. The test-obstacle regimens use only one command: straight forward at $0.8ms$.

Our controller evaluation mirrored tests with infants, so we used only the four obstacle terrains. Thus, terrains used in the obstacle training regimens are identical to those used in testing.

Cassie began each test trial facing the obstacle (slope, drop-off, gap, or bridge) at a distance of 3-3.5m, with a lateral offset of 0-0.25m. Each obstacle had a difficulty parameter, ranging from 0-1, randomized in training, that linearly adjusts the relevant property for each terrain. Cassie received 50 trials at each of 101 difficulty levels, for a total of 5050 trials per obstacle. The downward angle of slopes (1.5m long) ranged from 0-90\textdegree, the step-down height of drop-offs ranged from 0-1.5m, bridge width (1.5m long) ranged from 0.02-1.02m, and gap width ranged from 0-1m.

Training improved Cassie’s success at navigating obstacles, but performance varied depending on the obstacle (Fig. , top row). Notably, the single-test obstacle regimens (green curves) ensured greater success on every obstacle compared with the other regimens. On slopes and drop-offs, every training regimen improved success relative to the blind controller (gray curves), and on the drop-offs, the multi-test and combined regimens improved performance relative to the standard regimen (red curve). However, on gaps and bridges, results were mixed. On gaps, the standard, multi-test, and combined regimens performed equivalently to the blind controller. And on bridges—even at the lowest difficulty level—blind, standard, multi-test, and combined regimens produced success rates of . Yet, training to the test in the single-test obstacle regimen demonstrates that the gap and bridge obstacles were learnable.

Successful walking on more difficult obstacles was achieved in part by modifying gait after stepping on or over the obstacle (Fig. , second row)—that is, in the multiple steps on the slope or bridge and the single step to cross the drop-off or gap. On slopes, for example, the speed of the blind controller (gray curve) increased with difficulty as gravity pulled Cassie down steeper slopes; speed peaked at $\sim$30% difficulty and Cassie failed thereafter. The standard, multi-test, and combined training controllers also increased speed with difficulty, but speed peaked at $\sim$22% difficulty—before the peak of the blind controller—then decreased on steeper slopes as Cassie began using a braking strategy, resulting in success on steeper slopes than the blind controller could manage. The single-test training controller initially peaked even earlier at $\sim$11% difficulty, before implementing a braking strategy. The second peak for the single-test controller resulted from Cassie slipping down the slope, and speed decreased as Cassie began jumping down the slope at even more difficult increments.

On drop-offs and gaps, speed increased with difficulty. For drop-offs, increased speed likely reflects effects of gravity pulling the body down, but for gaps, it likely reflects Cassie launching its body to span larger gaps. Consistent with the poor success rate on bridges, Cassie increased speed on narrower bridges—making narrower bridges more challenging.

Although mid-obstacle gait modifications indicate that Cassie modified its gait to cope with more difficult obstacles, we cannot definitively categorize such adjustments as prospective. Increased speed to launch over a gap is produced before crossing (i.e., prospective), but decreased speed after stepping onto the slope could be in reaction to feeling the slant (i.e., reactive), and increased speed on the drop-off may be entirely out of Cassie’s control (i.e., neither prospective nor reactive). The best test of planning, therefore, is gait modifications prior to encountering the obstacle.

Cassie did not show compelling evidence of prospective speed adjustments prior to obstacles (Fig. , row 3). Speed and step length in the preceding two steps were constant or decreased only slightly across difficulty levels (e.g., from $\sim$0.83 m/s at 0 difficulty to $\sim$0.76 m/s at 50% difficulty).

Cassie did, however, show evidence of prospective gait modifications based on foot placement (Fig. , row 4). Cassie placed its last step prior to the obstacle closer to the edge as difficulty increased for the single-test regimen on slopes, drop-offs, and gaps, and for the standard, multi-test, and combined regimens on drop-offs and gaps. Placing the foot close to the edge is crucial, especially for drop-offs and gaps because it shortens the size of the step needed to cross. For drop-offs, the standard, multi-test, and combined regimen’s last step landed $\sim$0.15m from the edge at difficulty 0, but dropped to $\sim$0.08m on drop-offs at difficulty $\sim$25-100%.

Cassie showed markedly superior performance on each test obstacle when trained exclusively on that obstacle compared to the other training regimens. This indicates limited generalizability between these terrains, despite qualitative similarities. Hills are akin to slopes; ridges, blocks, and stairs are akin to drop-offs. Even the test obstacles approximate each other at some levels. The steepest slope is identical to the highest drop-off, and the widest gap approximates the gap around the bridge. The bridge obstacle is the most unique, requiring Cassie to not only handle the terrain in front of it, but to adjust navigation and guide foot placement to keep from falling off. We see from the poor single-test performance on bridges that it is also the most difficult. This may be explained somewhat by the reward function encouraging an uncompromising heading.

Interestingly, the success rates are scarcely improved from the standard regimen to the multi-test and combined regimens where Cassie is exposed to the test obstacles. The single-test controllers were trained for 20k iterations on the obstacle they would be evaluated on, whereas the controllers trained on the standard, multi-test obstacle, and combined standard and multi-test regimens were trained for 110k iterations spread across various terrains. The multi-test regimen therefore received 27.5k iterations on each test obstacle, but still shows inferior performance to single-test. This indicates that the variety of experience is hindering Cassie’s ability to learn the best strategies for each obstacle or possibly that longer training runs or larger models are required.

This interpretation is corroborated by the speeds mid-obstacle in Fig. , row 2. The single-test (green) curves stand apart from the others, indicating a different strategy was learned. This is most easily seen on slopes (column 1), where a braking behavior is adopted earliest (at a lower difficulty) by the single-test controller. Even more telling is the shift where it speeds up again, indicating a change in strategy not seen by the other controllers.

As discussed in Section , Cassie does not modulate its speed preceding an obstacle, but it does adjust foot placement, as illustrated in the last two rows of Fig. . The more the speed deviates from the commanded speed the more reward is lost, so an inflexible reward function may account for the consistent speeds. However, the intentional foot placement is most clearly seen on the gap obstacle in the bottom row of Fig. . Curiously, the single-test step lengths are larger and more similar to the blind controller’s, whereas the other regimens produce shorter preceding footsteps, possibly indicating a reluctance to step off the ledge. This combined with their low success rate indicates that they haven’t discovered an effective strategy (stepping across the gap) and are falling back on optimized failure modes.

Cassie demonstrated motor skills more advanced than any infant—jumping down high drop-offs and recovering from near-catastrophic falls. But we tested simulated Cassie. Real Cassie—with a physical body—would have required repairs after such feats. Babies also must deal with the consequences of errors, but cannot be taken to the shop for repairs. Instead, their body is built to cope with frequent errors in learning to walk: infants are small, low to the ground, and move slowly, decreasing impact forces produced by a fall [].

Moreover, every baby learns things Cassie did not learn: Infants use a wide range of prospective gait modifications, invent and use alternative strategies for obstacles where modifications are insufficient, and avoid crossing impossible obstacles []. Most critical, babies generalize learning. No infant experiences single-test training. To acquire behavioral flexibility and prospective control of locomotion, infants must generalize from everyday experiences. In this regard, any 18-month-old can run circles around Cassie. Infants’ greater flexibility and adaptability may result from more powerful learning mechanisms than pure RL. One hypothesis is that infants are “learning to learn”. That is, they learn to generate and gather the relevant information and use it to guide their actions from moment to moment.