Neural networks are parametric function approximators, that combine linear transformations $W$ and non-linear activation functions $f$ in alternating fashion:

where $\in l\{1,\dots ,L\}$. $h0$ corresponds to the input $x$ and $hL$ to an estimate of the target $^y$. Parameters of the model are commonly updated via gradient descent on a loss function, usually some form of negative log-likelihood. The power of neural networks stems from their ability to approximate any continuous function on a compact subset of $Rn$.

The task of learning model parameters $W$ can also be stated as a Bayesian inference problem:

for a given data set $=D\{(xi,yi)\}=i1N$, with inputs $=X\{xi\}=i1N$ and targets $=y\{yi\}=i1N$. Bayes’ theorem defines how we should update our beliefs as more information becomes available. As $\to N\infty$ the influence of the prior vanishes, while for $\to N0$ we can only rely on prior assumptions. In our context we assume that experience (i.e. $N$) increases with age and hence we use an approximation to Equation with data-sets of varying size to represent agents of different age.

Equation is in general hard to compute for models of useful complexity. Variational inference offers a tractable approximation to Equation ( ) . Let $q\varphi (W)$ be a distribution with parameters $\varphi$, that approximates the true posterior $p(|WD)$. Formulating the problem as a minimization of the Kullback-Leibler (KL) divergence between $q\varphi (W)$ and $p(|WD)$ leads to the evidence lower bound (ELBO):

which can be maximized with respect to $\varphi$ using standard optimization techniques.

In order to be able to scale to large data-sets Equation is often approximated using batches $\subseteq BD$ of size M, with the log-likelihood term being scaled appropriately:

Note, that only the first term of the ELBO depends on the data-set size $N$, while the second term is independent of it. Hence the divergence term will dominate for small data-sets, leading to models that closely reflect our prior assumptions. In this work we employ priors, that promote simple functions. Therefore, our models are able to capture successively more complex functions with increasing data-set size.

All models consist of $=L3$ layers with hidden layer sizes $=|hl|256$ and ELU activation functions ( ) , unless otherwise mentioned. Inputs $x$ correspond to flattened images of the scene and targets $y$ are dependent on the current task. We place a group horseshoe prior, which can be viewed as a continuous relaxation of a spike-and-slab prior ( ) , over all parameters:

and represent the approximate posterior $q\varphi (W)$ through a fully factorized distribution as proposed in ( ) . The sparsity hyperparameter of the horseshoe prior is fixed to $=\tau 010-5$. During training we approximate the expectation of the log-likelihood term with a single sample from $q\varphi (W)$ and make use of the local reparametrization trick ( ) . Gradient-based optimization is performed using Adam ( ) with batches consisting of $64$ samples. Results reported after training correspond to a Monte-Carlo estimate using 100 samples from $q\varphi (W)$.

The first task under investigation is concerned with occlusion events. It is based on an experiment conducted by \citeAbaillargeon2002acquisition. Each scene consists of a cylinder and a screen in form of a rectangular plane. During experimental manipulation the cylinder is moved back and forth behind the screen, while parts of the screen are removed. There are three different experimental conditions, each differing in which part of the screen is removed (top, bottom or everything removed). A depiction of the setup is shown in Figure (left). We are interested in infants’ ability to judge, whether the cylinder remains visible as it moves behind the screen, which is measured via violation-of-expectation methods ( ) . In violation-of-expectation methods gazing times for physically implausible events are measured. High gazing times for such events indicate, that children are surprised by the observation, which is interpreted as a violation of their expectation of what should have happened.

Empirical evidence indicates, that infants form an initial concept based on a behind/not-behind distinction starting from the age of 2.5 months. At this stage they have learned, that the cylinder remains visible while moving past the screen, if the entire middle section of the screen is removed. They do not expect the cylinder to be visible in the other two conditions. This initial concept is refined during later stages of development. At the age of three months they also predict to see parts of the cylinder, if only the bottom part of the screen is removed. The knowledge of 3.5 months olds additionally extends to screens with removed top fractions. Note, that the last condition is more challenging compared to the other two, as it involves a comparison of heights between the cylinder and the lower connection of the screen (if the connection height is lower than the cylinder, the cylinder will be visible, otherwise it will not be visible). Baillargeon et al. conclude from these observations, that infants start with initially simple rules about the laws of physics (behind/not-behind distinction), which become successively more sophisticated over time (reasoning about relative heights).

We construct an artificial version of this task as follows. Each input $x$ corresponds to a $\times 2424$ image with three channels, containing segregated information about the floor, the cylinder and the screen respectively. Input images show an initial configuration of the scene, in which the cylinder is located on the left side of the screen. Each target $y$ indicates the visibility of the cylinder when passing behind the screen, as it is moved towards the right side of the image. Cylinder height and position, screen position and size and floor level are determined randomly. We generate a training and test set, each consisting of 10000 data-points. Figure (center) shows examples for each of the conditions. Both sets include 2000 images for each of the three condition, as well as 4000 baseline images (nothing of the screen is removed). This is to ensure, that both $=y0$ (cylinder is visible) and $=y1$ (cylinder is not visible) are represented in equal fractions, i.e. the chance of guessing correctly is 50 percent.

We train otherwise identical models for $\in N\{256,512,1024,2048,8192\}$ until convergence and report results averaged over ten random seeds. The resulting performances on the artificial data-set are visualized in Figure (right). We observe, that the percentage of incorrect predictions, similar to the developmental progress in infants, decreases first in the easier conditions. The network is able to predict visibility of the cylinder reliably (with less than ten percent errors), if the entire middle section of the screen is removed, for $N$ larger than 512. For $N$ larger than 4096 it is additionally able to predict the correct targets, if the bottom part is removed (corresponding to knowledge of a three months old). This extends to the condition, in which the top part is removed for $=N8192$ (corresponding to knowledge of a 3.5 months old). Hence we conclude, that in this task the family of BNNs recovers the order of developmental stages observed in infants.

Next we take a closer look at infants’ knowledge of support events, for which we adopt another experiment of \citeAbaillargeon2002acquisition. In this task a scene consists of a box and a platform. The box is presented in different positions relative to the platform and the experimenter measures (via violation-of-expectation methods), if infants are able to predict, whether the given configuration is stable or not. Four different conditions are investigated. In the first the box is positioned either on top of the platform or some distance away from it. This condition requires a simple contact/no-contact distinction to make reliable predictions about the stability of the configuration. The second condition involves a distinction between different types of contact. Here the box connects with the platform either on the top (as before) or on the side. The third condition requires judgments based on the amount of contact, i.e. the box is only partially positioned on the platform. The final condition adds an additional layer of complexity, as it involves reasoning about non-rectangular shapes. The different conditions are summarized in Figure (left).

According to \citeAbaillargeon2002acquisition, from three months onward infants knowledge about the stability of a configuration is captured through a contact/no-contact distinction. At this stage they expect the box to be stable if and only if it touches the platform in some way. This initial hypothesis is than refined, as they grow older. At the age of around five months infants begin to distinguish between different types of contacts. They realize, that the box will only be stable, if it positioned above the platform, but not if it touches it on the side. Starting with an age of 6.5 months they are able to take into account the center of mass of simple objects (rectangular boxes) when reasoning about stability. This is extended to more complex, asymmetrical shapes at an age of roughly twelve months. As in the occlusion task, infants start with an initially simple hypothesis of how the laws of physics work, which is subsequently refined to better fit the observed data.

In our artificial version of this task inputs $x$ are represented as $\times 2424$ images with three channels (one for platform, box and floor) and targets $y$ are a binary indicator of the stability of the given configuration. Floor level as well as the size and position of the platform and the box are randomized. Again we generate a training set of 10000 samples and an equally large test set. In both sets 2500 data-points belong to each of the described conditions. Within each condition the amount of stable and unstable configurations is balanced, leading to a chance of 50 percent for guessing correctly. Example configurations are shown in Figure (center).

As before we train otherwise identical models for $\in N\{256,512,1024,2048,8192\}$ until convergence and report result averaged over ten random seeds. Inspecting Figure (right) we observe, that the family of BNNs first discovers solutions to the easier conditions (those where the box is positioned fully on the platform or on the side of it). If we increase $N$ to $4096$ or more, they are also able to reason reliably about stability in both of the center of mass conditions. Note, that the error rate decreases slower, when being exposed to the more complex, L-shaped objects, although this difference is only marginal. We conclude, that the models show pattern similar to the developmental progress of infants, as we increase the data-set size, akin to the observation from experiment 1, although not as pronounced.

Moving away from probing knowledge about physical laws, we next inspect a different domain: children’s intuitive counting abilities ( ) . A single trial consists of two images, each containing between 1 and 10 items, and the goal is to determine quickly, i.e. without to much deliberation time, which of the two images contains the larger quantity of items. The display time is adjusted depending on age, such that it is short enough to prevent serial counting. Objects within a trial are identical, but are selected from a set of different objects across trials. Two conditions either control for average item size or the summed continuous area. An example trial from the original task is shown in Figure (left).

Experimental data for this task has been obtained for three to six years olds, as well as for adults ( ) . Human perception is sensitive to the ratio between amounts of objects in the two images and not their difference, i.e. it follows a Weber-Fechner law. Levels of accuracy are measured for different ratios of objects, ranging from 1.11 (10:9) to 2.0 (2:1), from which Weber ratios are estimated. The Weber ratio is the smallest ratio, where a participant is able to identify the correct stimulus in more than 75 percent of the cases. Empirical results indicate, that the Weber ratio decreases during childhood and our ability to distinguish similar stimuli improves over time. Prior work ( ) estimates Weber ratios of around 1.53 in three years olds, which improves to 1.38 in four years olds, to 1.23 in five years olds, to 1.18 in six years olds and to 1.11 in adults. Overall the decrease of Weber ratios is well described through an exponential function of age (see Figure , right).

We created an artificial version of this task, with inputs $x$ corresponding to two $\times 2424$ images. For simplicity images contain only rectangular objects and the difference in quantity within a pair of images is always one. Targets $y$ are the number of objects in each images and the final prediction is obtained by comparing expected values between the two estimates. Object sizes are randomly drawn from $\{1,2,3\}$ and their position is randomized, such that neighbouring objects do not overlap. We controlled for an equal expected, total area between both images as done in the second condition of the original task. Both training and test set contain 6000 samples for each ratio from $\{$10:9, 9:8, 8:7, 7:6, 6:5, 5:4, 4:3, 3:2, 2:1$\}$. Examples are provided in Figure (center).

We train one model for each value from $\in N\cdot \{4,\dots ,15\}2048$. Hidden layers have 512 units in this experiment and we found it helpful to initialize weights for all models from a pretrained network. For the resulting models we calculate Weber ratios after estimating accuracy levels of 75 percent via linear interpolation and visualize the results in Figure (right). We observe an improvement of Weber ratios as the data-set size $N$ increases and conclude, that our models also follow a Weber-Fechner law. The resulting progression is well described through an exponential function 2 2 footnote 2 $=y+ae+-bxcd$, with $x$ corresponding to data-set size or age. of data-set size, a characteristic shared with the curve obtained from human participants. There is however a small gap between the overall model performance in the artificial task and that of human participants in the original one, even for large data-sets.