Learning Object Permanence from Video

Learning Object Permanence from Video Aviv Shamsian Ofri Kleinfeld^* Amir Globerson Gal Chechik equal contribution 1Bar-Ilan University, Ramat-Gan, Israel 1 1Bar-Ilan University, Ramat-Gan, Israel 1 2Tel Aviv University, Tel Aviv, Israel 2 1Bar-Ilan University, Ramat-Gan, Israel 1 3NVIDIA Research, Tel-Aviv, Israel https://chechiklab.biu.ac.il/~avivshamsian/OP/OP_HTML.html 3

Object Permanence allows people to reason about the location of non-visible objects, by understanding that they continue to exist even when not perceived directly. Object Permanence is critical for building a model of the world, since objects in natural visual scenes dynamically occlude and contain each-other. Intensive studies in developmental psychology suggest that object permanence is a challenging task that is learned through extensive experience.

Here we introduce the setup of learning Object Permanence from labeled videos. We explain why this learning problem should be dissected into four components, where objects are (1) visible, (2) occluded, (3) contained by another object and (4) carried by a containing object. The fourth subtask, where a target object is carried by a containing object, is particularly challenging because it requires a system to reason about a moving location of an invisible object. We then present a unified deep architecture that learns to predict object location under these four scenarios. We evaluate the architecture and system on a new dataset based on CATER, and find that it outperforms previous localization methods and various baselines.

1 section 1 1 §1 <tag close=" ">1</tag>Introduction

Understanding dynamic natural scenes is often challenged by objects that contain or occlude each other. To reason correctly about such visual scenes, systems need to develop a sense of Object Permanence (OP) []. Namely, the understanding that objects continue to exist and preserve their physical characteristics, even if they are not perceived directly. For example, we want systems to learn that a pedestrian occluded by a truck may emerge from its other side, but that a person entering a car would “disappear” from the scene.

The concept of OP received substantial attention in the cognitive development literature. Piaget hypothesized that infants develop OP relatively late (at two years of age), suggesting that it is a challenging task that requires deep modelling of the world based on sensory-motor interaction with objects. Later evidence showed that children learn OP for occluded targets early []. Still, only at a later age do children develop understanding of objects that are contained by other objects []. Based on these experiments we hypothesize that reasoning about the location of non-visible objects may be much harder when they are carried inside other moving objects.

Figure 1 Figure 1 1 Figure 1

(a) (b) (c) (d)

1Inferring object location in rich dynamic scenes involves four different tasks, and two different types of reasoning. (a) The target, a red ball, is fully visible. (b) The target is fully-or partially occluded by the static cube. (c) The target is located inside the cube and fully covered. (d) The non-visible target is located inside another moving object; its location changes even though it is not directly visible Figure 1Inferring object location in rich dynamic scenes involves four different tasks, and two different types of reasoning. (a) The target, a red ball, is fully visible. (b) The target is fully-or partially occluded by the static cube. (c) The target is located inside the cube and fully covered. (d) The non-visible target is located inside another moving object; its location changes even though it is not directly visible

Reasoning about the location of a target object in a video scene involves four different subtasks of increasing complexity (Figure ). These four tasks are based on the state of the target object, depending if it is (1) visible, (2) occluded, (3) contained or (4) carried. The visible case is perhaps the simplest task, and corresponds to object detection, where one aims to localize an object that is visible. Detection was studied extensively and is viewed as a key component in computer vision systems. The second task, occlusion, is to detect a target object which becomes transiently invisible by a moving occluding object (e.g., bicycle behind a truck). Tracking objects under occlusion can be very challenging, especially with long-term occlusions [].

Third, in a containment scenario, a target object may be located inside another container object and become non-visible [] (e.g., a person enters a store). Finally, the fourth case of a carried object is arguably the most challenging task. It requires inferring the location of a non-visible object located inside a moving containing object (e.g., a person enters a taxi that leaves the scene). Among the challenging aspects of this task is the need to keep a representation of which object should be tracked at every time point and the need to “switch states” dynamically through time. This task received little attention in the computer vision community so far.

We argue that reasoning about the location of a non-visible object should address two distinct and fundamentally different cases: occlusion and containment. First, to localize an occluded object, an agent has to build an internal state that models how the object moves. For example, when we observe a person walking in the street, we can predict her ever-changing location even if occluded by a large bus. In this mode, our reasoning mechanism keeps attending to the person and keeps inferring her location from past data. Second, localizing contained objects is fundamentally different. It requires a reasoning mechanism that switches to attend to the containing object, which is visible. Here, even though the object of interest is not-visible, its location can be accurately inferred from the location of the visible containing object. We demonstrate below that incorporating these two reasoning mechanisms leads to more accurate localization in all four subtasks.

Specifically, we develop a unified approach for learning all four object localization subtasks in video. We design a deep architecture that learns to localize objects that may be visible, occluded, contained or carried. Our architecture consists of two reasoning modules designed to reason about (1) carried or contained targets, and (2) occluded or visible targets. The first reasoning component is explicitly designed to answer the question “Which object should be tracked now?”. It does so by using an LSTM to weight the perceived locations of the objects in the scene. The second reasoning component leverages the information about which object should be tracked and previous known locations of the target to localize the target, even if it is occluded. Finally, we also introduce a dataset that is based on videos from CATER [], enriched with new annotations about task type and about ground-truth location of all objects.

Our main novel contributions are: (1) We conceptualize that localizing non-visible objects requires two types of reasoning: about occluded objects and about carried ones. (2) We define four subtypes of localization tasks and introduce annotations for the CATER dataset to facilitate evaluating each of these subtasks. (3) We describe a new unified architecture for all four subtasks, which can capture the two types of reasoning, and we show empirically that it outperforms multiple strong baselines.

2 section 2 2 §2 <tag close=" ">2</tag>Related Work 2.1 subsection 2.1 2.1 §2.1 <tag close=" ">2.1</tag>Relational Reasoning in Synthetic Video Datasets

Recently, several studies provided synthetic datasets to explore object interaction and reasoning. Many of these studies are based on CLEVR [], a synthetic dataset designed for visual reasoning through visual question answering. CLEVRER [] extended CLEVR to video, focusing on the causal structures underlying object interactions. It demonstrated that visual reasoning models that thrive on perception based tasks often perform poorly in causal reasoning tasks.

Most relevant for our paper, CATER [] is a dataset for reasoning about object action and interactions in video. One of the three tasks defined in CATER, the snitch localization task, is closely related to the OP problem studied here. It is defined as localizing a target at the end of a video, where the target is usually visible. Our work refines their setup, learning to localize the target through the full video, and breaks down prediction into four types of localization tasks. As a result, we provide a fine-grained insight about the architectures and reasoning that is required for solving the complex localization task.

2.2 subsection 2.2 2.2 §2.2 <tag close=" ">2.2</tag>Architectures for Video Reasoning

Several recent papers studied the effectiveness of CNN-based architectures for video action recognition. Many approaches use 3D convolutions for spatiotemporal feature learning [] and separate the spatial and temporal modalities by adding optical flow as a second stream []. These models are computationally expensive because 3D convolution kernels may be costly to compute. As a result, they may limited to sequence length to 20-30 frames []. In [] it was proposed to sparsely sample video frames to capture temporal relations in action recognition datasets. However, sparse sampling may be insufficient for long occlusion and containment sequences, which is the core of our OP focus.

Another strategy for temporal aggregation is to use recurrent architectures like LSTM [], connecting the underlying CNN output along the temporal dimension []. [] combined LSTM with spatial attention, learning to attend to those parts of the video frame that are relevant for the task as the video progresses. In Section we experiment with a spatial attention module, which learns to dynamically focus on relevant objects.

2.2.1 subsubsection 2.2.1 2.2.1 §2.2.1 <tag close=" ">2.2.1</tag>Tracking with Object Occlusion.

A large body of work has been devoted to tracking objects []. For objects under complex occlusion like carrying, early work studied tracking using classical techniques and without deep learning methods. For instance, [] used the idea of object permanence to track objects through long-term occlusions. They located objects using adaptive appearance models, spatial distributions and inter-occlusion relationships. In contrast, the approach presented in this paper focuses on a single deep differentiable model to learn motion reasoning end-to-end. [] succeeds to track occluded targets by learning how their movement is coupled with the movement of other visible objects. Unfortunately, the dataset studied here, CATER [], has weak object-object motion coupling by design. Specifically, when measuring the correlation between the movement of the target and other object (as in []), we found that the correlation in 94% of the videos was not statistically significant.

More recently, models based on Siamese neural network achieved SOTA results in object tracking []. Despite the power of these architectures, tracking highly-occluded objects is still challenging []. The tracker of [], DaSiamRPN, extends the region-proposal sub-network of []. It was designed for long-term tracking and handles full occlusion or out-of-view scenarios. DaSiamRPN was used as a baseline for the snitch localization task in CATER [], and we evaluated its performance for the OP problem in Section .

2.2.2 subsubsection 2.2.2 2.2.2 §2.2.2 <tag close=" ">2.2.2</tag>Containment.

Few recent studies explored the idea of containment relations. [] recovered incomplete object trajectories by reasoning about containment relations. [] proposed an unsupervised model to categorize spatial relations, including containment between objects. The containment setup defined in these studies differs from the one defined here in that the contained object is always at least partially visible [], or the containment does not involve carrying [].

3 section 3 3 §3 <tag close=" ">3</tag>The Learning Setup: Reasoning about Non-Visible Objects

We next formally define the OP task and learning setup. We are given a set of videos $v 1, …, v N$ where each frame $x i t$ in video $v i$ is accompanied by the bounding box position $B i t$ of the target object as its label. The goal is to predict for each frame a bounding box $^B i t$ of the target object that is closest (in terms of $L 1$ distance) to the ground-truth bounding box $B i t$ .

We define four localization tasks: (1) Localizing a visible object, which we define as an object that is at least partially visible. (2) Localizing an occluded object, which we define as an object that is fully occluded by another object. (3) Localizing an object contained by another object, thus also completely non visible. (4) Localizing an object that is carried along the surface by a containing object. Thus in this case the target is moving while being completely non-visible. Together, these four tasks form a localization task that we call object-permanence localization task, or OP.

In Section , we also study a semi-supervised learning setup, where at training time the location $B i t$ of the target is provided only in frames when it is visible. This would correspond to the case of a child learning object permanence without explicit feedback about where an object is located when it is hidden.

It is instructive to note how the task we address here differs from the tasks of relation or reaction recognition []. In these tasks, models are trained to output an explicit label that captures the name of the interaction or relation (e.g., “behind”, “carry”). In our task, the model aims to predict the location of the target (a regression problem), but it is not trained to name it explicitly (occluded, contained). While it is possible that the model creates some implicit representation describing the visibility type, this is not mandated by the loss or the architecture.

Figure 2 Figure 2 2 Figure 2 2The architecture of Object Permanence network (OPNet) consists of three components. (a) Perception module for detection. (b) Reasoning module for inferring which object to track in case the target is carried or contained. (c) A second reasoning module for occluded or visible targets, and for refining the exact location of the predicted target. Figure 2The architecture of Object Permanence network (OPNet) consists of three components. (a) Perception module for detection. (b) Reasoning module for inferring which object to track in case the target is carried or contained. (c) A second reasoning module for occluded or visible targets, and for refining the exact location of the predicted target.

4 section 4 4 §4 <tag close=" ">4</tag>Our Approach

We describe a deep network architecture designed to address the four localization subtasks of the OP task. We refer to the architecture as OPNet. It contains three modules, that account for the perception and inference computations which facilitate OP (see Figure ).

Perception and detection module (Figure a): A perception module, responsible for detecting and tracking visible objects. We incorporated a Faster R-CNN [] object detection model, fine-tuned on frames from our dataset, as the perception component of our model. After pre-training, we used the detector to output the bounding boxes together with identifiers of all objects in any given frame. Specifically, we represent a frame using a $× K 5$ matrix. Each row in the matrix represents an object using $5$ values: four values of the bounding box ( $x 1, y 1, x 2, y 2$ ) and one visibility bit, which indicates whether the object is visible or not. As the video progresses, we assign a unique row to each newly identified object. If an object is not detected in a given frame, its corresponding information (assigned row) is set to zero. In practice, $= K 15$ was the maximal number of objects in a single video in our dataset. Notably, the videos in the dataset we used do not contain two identical objects, but we found that the detector sometimes mistakes one object for another.

“Who to track?” module (Figure c): responsible for understanding which object is currently covering the target. This component consists of a single LSTM layer with a hidden dimension of 256 neurons and a linear projection matrix. After applying the LSTM to the object bounding boxes, we project its output to $K$ neurons, each representing a distinct object in the frame. Finally we apply a softmax layer, resulting in a distribution over the objects in the frame. This distribution can be viewed as an attention mask focusing on the object that covers the target in this frame. Importantly, we do not provide explicit supervision to this attention mask (e.g., by explicitly “telling the model” during training what is the correct attention mask). Rather, our only objective is the location of the target. The output of this module is $5$ numbers per frame. It is computed as the the weighted average over the $× K 5$ outputs of the previous stage, weighted by the attention mask.

“Where is it” module (Figure b): learns to predict the location of occluded targets. This final component consists of a second LSTM and a projection matrix. Using the output of the previous component, this component is responsible for predicting the target localization. It takes the output of the previous step ( $5$ values per frame), feeds it into the LSTM and projects its output to four units, representing the predicted bounding box of the target for each frame.

5 section 5 5 §5 <tag close=" ">5</tag>The LA-CATER Dataset

To train models and evaluate their performance on the four OP subtasks defined above, we introduce a new set of annotations to the CATER dataset []. We refer to these as Localization Annotations (LA-CATER).

The CATER dataset consists of 5,500 videos generated programmatically using the Blender 3D engine. Each video is 10-second long (300 frames) and contains 5 to 10 objects. Each object is characterized by its shape (cube, sphere, cylinder and inverted cone), size (small, medium, large), material (shiny metal and matte rubber) and color (eight colors). Every video contains a golden small sphere referred to as “the snitch”, that is used as the target object which needs to be localized.

For the purpose of this study, we generated videos following a similar configuration to the one used by CATER, but we computed additional annotations during video generation. Specifically, we augmented the CATER dataset with ground-truth bounding boxes locations of all objects. These annotations were programmatically extracted from the Blender engine, by projecting the internal 3D coordinates of objects are to the 2D pixel space.

We further annotated videos with detailed frame-level annotations. Each frame was labeled with one of four classes: visible, fully occluded, contained (i.e., covered, static and non-visible) and carried (i.e., covered, moving and non-visible). This classification of frames matches the four localization subtasks of the OP problem. To compute these annotations, we computed the line-of sight from the camera position to determine if a target is occluded by another object, or occluding it.

LA-CATER includes a total number of 14K videos split into train, dev and test datasets. See Table for a classification of video frames to each one of the localization subtasks across the dataset splits. Further details about dataset preparation are provided in appendix .

Table 1 Table 1 1 Table 1 1Fraction of frames per type in the train, dev and test sets of LA-CATER. Occluded and carried target frames make up less than 8% of the frames, but they present the most challenging prediction tasks.

Table 1Fraction of frames per type in the train, dev and test sets of LA-CATER. Occluded and carried target frames make up less than 8% of the frames, but they present the most challenging prediction tasks.

Number of
Frames

Visible

Occluded

Contained

Carried
Train	9,300	63.00%	3.03%	29.43%	4.54%
Dev	3,327	63.27%	2.89%	29.19%	4.65%
Test	1,371	64.13%	3.07%	28.56%	4.24%

6 section 6 6 §6 <tag close=" ">6</tag>Experiments

We describe our experimental setup, compared methods and evaluation metrics. Implementation details are given in Appendix .

6.1 subsection 6.1 6.1 §6.1 <tag close=" ">6.1</tag>Baselines and Model Variants

We compare our proposed OPNet with six other architectures designed to solve the OP tasks. Since we are not aware of previously published unified architectures designed to solve all OP tasks at once, we used existing models as components in our baselines. All baseline models receive the predictions of the object detector (perception) component as their input.

(A) Programmed Models. We evaluated two programmed models. These models are “hard-coded” rather than learned. They are designed to reflect models that programmatically solve the reasoning task.

• item 1st item

(1) Detector + Tracker. Using the detected location of the target, this method initiates a DaSiamRPN tracker [] to track the target. Whenever the target is no longer visible, the tracker is re-initiated to track the object located in the last known location of the target.

• item 2nd item

(2) Detector + Heuristic. When the target is not detected, the model switches from tracking the target to tracking the object located closest to last known location of the target. The model also employs an heuristic logic to adjust between the sizes of the current tracked object and the original target.

(B) Learned Models. We evaluated four learned baselines with an increasing level of representation complexity.

• item 1st item

(3) OPNet. The proposed model, as presented in Section .

• item 2nd item

(4) Baseline LSTM. This model uses a single unidirectional LSTM layer with a hidden state of 512 neurons, operating on the temporal (frames) dimension. The input to the LSTM is the concatenation of the objects input representations. It is the simplest learned baseline as the input representation is not transformed non-linearly before being fed to the LSTM.

• item 3rd item

(5) Non-Linear + LSTM. This model augments the previous model and increases the complexity of the scene representation. The input representations are upsampled using a linear layer followed by a ReLU activation, resulting in a 256-dimensional vector representation for each object in the frame. These high-dimensional objects representations are concatenated and fed into the LSTM.

• item 4th item

(6) Transformer + LSTM. This model augments the previous baselines by introducing a much complex representations for objects in frame. We utilized a transformer encoder [] after up-sampling the input representations, employing self attention between all the objects in a frame. We used a transformer encoder with 2 layers and 2 attention heads, yielding a single vector containing the target attended values. These attended values, which corresponds to each other object in the frame, are then fed into the LSTM.

• item 5th item

(7) LSTM + MLP. This model (Figure ) ablates the second LSTM module (c) in the model presented in Section .

6.2 subsection 6.2 6.2 §6.2 <tag close=" ">6.2</tag>Evaluation Metric

We evaluate model performance at a given frame $t$ by comparing the predicted target localization and the ground truth (GT) target localization. We use two metrics as follows. First, the intersection over union (IoU) metric.

(1) Equation 1 1

= ⁢ I o U t ∩ B ⁢ G T t B p t ∪ B ⁢ G T t B p t,

where $B p t$ denotes the predicted bounding box for frame $t$ and $B ⁢ G T t$ denotes the ground truth bounding box for frame $t$ .

Second, we evaluate models using the mean average precision (MAP) metric. MAP is computed by employing an indicator function on each frame, determining whether the IoU value is greater than a predefined threshold, then averaging across frames in a single video and all the videos in the dataset.

(2) Equation 2 2

= ⁢ A P ⁢ 1 n ∑ = t 1 n 1 t = ⁢, where 1 t {1 > ⁢ I o U t ⁢ I o U t h r e s h o l d 0 otherwise

(3) Equation 3 3

= ⁢ M A P ⁢ 1 N ∑ = v 1 N ⁢ A P v .

These per-frame metrics allow us to quantify the performance on each of the four OP subtasks separately.

7 section 7 7 §7 <tag close=" ">7</tag>Results

We start with comparing OPNet with the baselines presented in Section . We then provide more insights into the performance of the models by repeating the evaluations with “Perfect Perception” in Section . Section describes a semi-supervised setting of training with visible frames only. Finally, in Section we compare OPNet with the models presented in the CATER paper on the original CATER data.

Table 2 Table 2 2 Table 2 2Mean IoU performance of various models on the LA-CATER test data. “

±

” denotes the standard error of the mean (SEM). OPNet performs consistently well across all subtasks. Also, on the contained and carried frames OPNet is significantly better than the other methods.

Table 2Mean IoU performance of various models on the LA-CATER test data. “ $±$ ” denotes the standard error of the mean (SEM). OPNet performs consistently well across all subtasks. Also, on the contained and carried frames OPNet is significantly better than the other methods.

Mean IoU $±$ SEM

Visible

Occluded

Contained

Carried
Overall
Detector + Tracker	90.27 $± 0.13$	53.62 $± 0.58$	39.98 $± 0.38$	34.45 $± 0.40$	71.23 $± 0.51$
Detector + Heuristic	90.06 $± 0.14$	47.03 $± 0.73$	55.36 $± 0.53$	55.87 $± 0.59$	76.91 $± 0.43$
Baseline LSTM	81.60 $± 0.19$	59.80 $± 0.61$	49.18 $± 0.64$	21.53 $± 0.40$	67.20 $± 0.53$
Non-Linear + LSTM	88.25 $± 0.14$	70.14 $± 0.62$	55.66 $± 0.67$	24.58 $± 0.44$	73.53 $± 0.51$
Transformer + LSTM	90.82 $± 0.14$	80.40 $± 0.61$	70.71 $± 0.78$	28.25 $± 0.45$	80.27 $± 0.50$
OPNet (LSTM + MLP)	88.11 $± 0.16$	55.32 $± 0.85$	65.18 $± 0.89$	57.59 $± 0.85$	78.85 $± 0.52$
OPNet (LSTM + LSTM)	88.89 $± 0.16$	78.83 $± 0.56$	76.79 $± 0.62$	56.04 $± 0.77$	81.94 $± 0.41$

We first compare OPNet and the baselines presented in Section . Table shows IoU for all models in all four sub-tasks and Figure presents the MAP accuracy of the models across different IoU thresholds.

It can be seen in Table that OPNet performs consistently well across all subtasks and outperforms all other models overall. On the visible and occluded frames performance is similar to other baselines. But on the contained and carried frames, OPNet is significantly better than the other methods. This is likely due to OPNet’s explicit modeling of the object to be tracked.

Table also reports results for two variants of OPNet: OPNet (LSTM+MLP) and OPNet (LSTM+LSTM). The former is missing the second module (“Where is it” in Figure ) which is meant to handle occlusion and indeed under-performs for occlusion frames (the “occluded” and “contained” subtasks). This highlights the importance of using the two LSTM modules in Figure .

Figure provides interesting insight into the behavior of the programmed models (namely Detector + Tracker and Detector + Heuristic). It can be seen that these models perform well when the IoU threshold is low. This reflects the fact that they have a good coarse estimate of where the target is, but fail to provide more accurate localization. On the other hand our OPNet model does well for accurate localization, presumably due to its learned “Where is it” module.

Figure 3 Figure 3 3 Figure 3 3Mean average precision (MAP) as a function of IoU thresholds. The two programmed models, Detector+Tracker (blue) and Detector+Heuristic (orange) perform well when the IoU threshold is low, providing a good coarse estimate of target location. OPNet performs well on all subtasks. Figure 3Mean average precision (MAP) as a function of IoU thresholds. The two programmed models, Detector+Tracker (blue) and Detector+Heuristic (orange) perform well when the IoU threshold is low, providing a good coarse estimate of target location. OPNet performs well on all subtasks. 7.1 subsection 7.1 7.1 §7.1 <tag close=" ">7.1</tag>Reasoning with Perfect Perception

The OPNet model contains an initial “Perception” module that analyzes the frame pixels to get bounding boxes. Errors in this component will naturally propagate to the rest of the model and adversely affect results. Here we analyze the effect of the perception module by replacing it with ground truth bounding boxes and visibility bits. See Appendix for details on extracting ground-truth annotations. In this setup all errors reflect failure in the reasoning components of the models.

Table 3 Table 3 3 Table 3 3Mean IoU performance with the Perfect Perception setup. “

±

” denotes the standard error of the mean (S.E.M.). Results are similar in nature to those with imperfect, detector-based, perception (Table ). All models improve when using the ground-truth perception information. The subtask that improves the most with OPNet is the carried task.

Table 3Mean IoU performance with the Perfect Perception setup. “ $±$ ” denotes the standard error of the mean (S.E.M.). Results are similar in nature to those with imperfect, detector-based, perception (Table ). All models improve when using the ground-truth perception information. The subtask that improves the most with OPNet is the carried task.
Mean IoU $±$ SEM	Visible	Occluded	Contained	Carried	Overall
DETECTOR + TRACKER	90.27 $± 0.13$	53.62 $± 0.58$	39.98 $± 0.38$	34.45 $± 0.40$	71.23 $± 0.51$
DETECTOR + HEURISTIC	95.59 $± 0.34$	30.40 $± 0.81$	59.81 $± 0.47$	59.33 $± 0.50$	81.24 $± 0.49$
BASELINE LSTM	75.22 $± 0.31$	50.52 $± 0.75$	45.10 $± 0.62$	19.12 $± 0.36$	61.41 $± 0.53$
NON-LINEAR + LSTM	88.63 $± 0.25$	65.73 $± 0.82$	58.77 $± 0.70$	23.89 $± 0.41$	74.53 $± 0.54$
TRANSFORMER + LSTM	93.99 $± 0.24$	81.31 $± 0.88$	75.75 $± 0.85$	28.01 $± 0.44$	83.78 $± 0.55$
OPNet (LSTM + MLP)	88.11 $± 0.16$	19.39 $± 0.60$	77.40 $± 0.68$	78.25 $± 0.65$	83.84 $± 0.48$
OPNet (LSTM + LSTM)	88.78 $± 0.25$	67.79 $± 0.69$	83.47 $± 0.47$	76.42 $± 0.66$	85.44 $± 0.38$

Table provides the IoU performance and Figure the MAP for all compared methods on all four subtasks. The results are similar to the previous results. When compared to the previous section (imperfect, detector-based, perception), the overall trend is the same, but all models improve when using the ground truth perception information. Interestingly, the subtask that improves the most from using ground truth boxes is the carried task. This makes sense, since it is the hardest subtask and the one that most relies on having the correct object locations per frame.

Figure 4 Figure 4 4 Figure 4 4Mean average precision (MAP) as a function of IoU thresholds for reasoning with Perfect Perception (Section ). The most notable performance gain of OPNet (pink and brown curves) was with carried targets (subtask d). Figure 4Mean average precision (MAP) as a function of IoU thresholds for reasoning with Perfect Perception (Section ). The most notable performance gain of OPNet (pink and brown curves) was with carried targets (subtask d). 7.2 subsection 7.2 7.2 §7.2 <tag close=" ">7.2</tag>Learning only from Visible Frames

We now examine a learning setup in which localization supervision is available only for frames where the target object is visible. This setup corresponds more naturally to the process by which people learn object permanence. For instance, imagine a child learning to track a carried (non visible) object for the first time and receiving a surprising feedback only when the object reappears in the scene.

In absence of any supervision when the target is non-visible, incorporating an extra auxiliary loss is needed to account for these frames. Towards this end, we incorporated an auxiliary consistency loss that minimizes the change between predictions in consecutive frames. $= L ⁢ c o n s i s t e n c y ⁢ 1 n ∑ = t 1 n ∥ - b t b - t 1 ∥ 2$ . The total loss is defined as an interpolation between the localization loss and the consistency loss, balancing their different scales: $= L + ⋅ α L ⁢ l o c a l i z a t i o n ⋅ β L ⁢ c o n s i s t e n c y$ Details on choosing the values of $α$ and $β$ are provided in the supplementary.

Table 4 Table 4 4 Table 4 4IoU performance for the only visible supervision setting. “

±

” denote the standard error of the mean (S.E.M.). The models perform well when the target is visible, fully occluded or contained without movement, but not when the target is carried.

Table 4IoU performance for the only visible supervision setting. “ $±$ ” denote the standard error of the mean (S.E.M.). The models perform well when the target is visible, fully occluded or contained without movement, but not when the target is carried.

Mean IoU

Visible

Occluded

Contained

Carried
Overall
Baseline LSTM	88.61 $± 0.16$	80.39 $± 0.54$	68.35 $± 0.76$	27.39 $± 0.45$	78.09 $± 0.49$
Non Linear + LSTM	89.30 $± 0.15$	82.49 $± 0.45$	67.25 $± 0.75$	27.34 $± 0.45$	78.15 $± 0.49$
Transformer + LSTM	88.33 $± 0.15$	83.74 $± 0.44$	69.93 $± 0.77$	27.65 $± 0.54$	78.43 $± 0.49$
OPNet (LSTM + MLP)	88.45 $± 0.17$	48.03 $± 0.82$	10.95 $± 0.51$	7.28 $± 0.30$	61.18 $± 0.69$
OPNet (LSTM + LSTM)	88.95 $± 0.16$	81.84 $± 0.48$	69.01 $± 0.76$	27.50 $± 0.45$	78.50 $± 0.49$

Table shows the mean IoU for this setup (compare with Table ). The baselines perform well when the target is visible, fully occluded or contained without movement. This phenomenon goes hand-in-hand with the inductive bias of the consistency loss. Usually, to solve these subtasks, a model only needs to predict the last known target location. This explains why the OPNet (LSTM+MLP) model performs so poorly in this setup.

We note that the performance of non-OPNet models on the carried task is similar to that obtained using full supervision (see Table , Section ) . This suggests that these models fail to use the supervision for the “carried” task, and further reinforces the observation that localizing carried object is highly challenging.

7.3 subsection 7.3 7.3 §7.3 <tag close=" ">7.3</tag>Comparison with CATER Data

The original CATER paper [] considered the “snitch localization” task, aiming to localize the snitch at the last frame of the video, and formalized as a classification problem. The x-y plane was divided with a 6-by-6 grid, and the goal was to predict the correct cell of that grid.

Here we report the performance of OPNet and relevant baselines evaluated on the exact setup as in [], to facilitate comparison between our models and the results reported there. Table shows the accuracy and $L 1$ distance metrics for this evaluation. OPNet significantly improves over all baselines from []. It cuts down the classification error from $% 40$ error down to $% 24$ , and the $l 1$ distance from $1.2$ to $0.54$ .

Table 5 Table 5 5 Table 5 5Classification accuracy on the CATER dataset using the metrics of []. OPNet significantly improves over all baselines for the “snitch localization task”.

Table 5Classification accuracy on the CATER dataset using the metrics of []. OPNet significantly improves over all baselines for the “snitch localization task”.
	Accuracy	$L 1$ Distance
Model	(higher is better)	(lower is better)
DaSiamRPN	33.9	2.4
TSN-RGB + LSTM	25.6	2.6
R3D + LSTM	60.2	1.2
OPNet (Ours)	74.8	0.54

Figure 5 Figure 5 5 Figure 5

(a) (b)

5Screenshots from the model comparison video files. Blue boxes denote the ground truth location. Yellow boxes denote the predicted location. OPNet (ours) is at the bottom right panel. (a) The target is contained and then carried by the blue cone and is captured successfully by OPNet. (b) The target is occluded by the red cone and purple ball. These occlusions confuse all baselines, while OPNet localizes the target accurately. Figure 5Screenshots from the model comparison video files. Blue boxes denote the ground truth location. Yellow boxes denote the predicted location. OPNet (ours) is at the bottom right panel. (a) The target is contained and then carried by the blue cone and is captured successfully by OPNet. (b) The target is occluded by the red cone and purple ball. These occlusions confuse all baselines, while OPNet localizes the target accurately.

Figure 6 Figure 6 6 Figure 6

(a) (b) (c) (d) (e)

6Examples of a success case (top row) and a failure case (bottom row) for localizing a carried object. The blue box marks the ground-truth location. The yellow box marks the predicted location. Top (a) The target object is visible; (b-c) The target becomes covered and carried by the orange cone; (d-e) The big golden cone covers and carries the orange cone, illustrating recursive containment. The target object is not visible, but OPNet successfully tracks it. Bottom (c-d) OPNet accidentally switches to the wrong cone object (the yellow cone instead of the brown cone); (e) OPNet correctly finds when the yellow cone is picked up and switches to track the blue ball that was underneath. Figure 6Examples of a success case (top row) and a failure case (bottom row) for localizing a carried object. The blue box marks the ground-truth location. The yellow box marks the predicted location. Top (a) The target object is visible; (b-c) The target becomes covered and carried by the orange cone; (d-e) The big golden cone covers and carries the orange cone, illustrating recursive containment. The target object is not visible, but OPNet successfully tracks it. Bottom (c-d) OPNet accidentally switches to the wrong cone object (the yellow cone instead of the brown cone); (e) OPNet correctly finds when the yellow cone is picked up and switches to track the blue ball that was underneath. 7.4 subsection 7.4 7.4 §7.4 <tag close=" ">7.4</tag>Qualitative Examples

To gain insight into the successes and failures of our model, we now analyze specific examples. We provide two sets of examples to illustrate: (1) Comparison between baselines and variants over the same set of videos; (2) Wins and losses of our approach.

Model Comparison. We show two videos comparing OPNet with baselines and other variants. In both videos, four competing methods are applied to the same video scene. We recommend playing videos at a slow speed.

Figure 7 Figure 7 7 Figure 7

(a) (b)

7Switching attention across objects. In each pair of panels, each row traces the probability assigned to an object along the video in the ground truth (left) and predicted attention (right). (a) The system successfully switches attention from object 1 (target) when it is contained by object 6 and then carried by object 3. (b) After a successful switch from the object 1 to 10, the system incorrectly witches to object 3. Figure 7Switching attention across objects. In each pair of panels, each row traces the probability assigned to an object along the video in the ground truth (left) and predicted attention (right). (a) The system successfully switches attention from object 1 (target) when it is contained by object 6 and then carried by object 3. (b) After a successful switch from the object 1 to 10, the system incorrectly witches to object 3. • item 1st item

The first model comparison video (https://youtu.be/TZgoxoKcGrE) shows one visual scene analyzed by four methods. OPNet (ours) successfully localizes the target throughout the video. When the target is “carried”, the Transformer model (bottom left) fails to switch and instead of tracking the carrying object it keeps predicting the last seen location of the target. The Tracker model (top left) switches to a wrong object. The Heuristic model (top right) successfully tracks the object containing the target, adjusting well to the target size. See Figure (a).

• item 2nd item

The second model comparison video (https://youtu.be/KoxbhgalazU) shows a visual scene analyzed by four methods. In this video, the target is being occluded by multiple objects, including full occlusion, which makes it challenging to track. The Tracker, Heuristic and OPNet MLP models occasionally drift from the target when it is fully occluded by a large object. OPNet (ours) successfully localizes the target throughout the video. See Figure (b).

Wins and Losses of OPNet. We provide interesting examples of OPNet success and failures, adding insights into the behaviour and limitations of the OPNet model.

• item 1st item

The video https://youtu.be/FnturB2Blw8 provides a “win” example. It demonstrates the power of OPNet and its “who to track” reasoning component. In the video, the model handles phases of recursive containment (“babushka”), which involve “carrying”. It suggests that OPNet learns an implicit representation of the object actions (pick-up, slide, contain etc.) even though it was not explicitly trained to do so. See Figure (top row)

• item 2nd item

The video https://youtu.be/qkdQSHLrGqI illustrates a failure of our model. It shows an example where OPNet fails to switch between tracked objects when the target is “carried”. The model accidentally switches to a wrong cone object (the yellow cone) that already contains another object, not the target. Interestingly, OPNet properly identifies when the yellow cone is picked up and switches to track the blue ball that was contained by the yellow cone. It suggests that OPNet has implicitly learned the “meaning” of actions performed by objects, without being explicitly trained to do so. See Figure (bottom row)

Further insight may be provided by comparing the attention mask of the OPNet “Who to Track” module and the ground-truth mask of the containing or carrying object. Figure compares these masks for success and failure cases. It can be seen that OPNet nicely tracks the correct object for most of the frames.

8 section 8 8 §8 <tag close=" ">8</tag>Conclusion

We considered the problem of localizing one target object in a highly dynamic scenes where the object can be occluded, contained or even carried away by another object. We name this task object permanence, following the cognitive concept of a target object that is physically present in a scene but is occluded and carried in various ways. We presented an architecture called OPNet, whose components correspond to the natural perceptual and reasoning stages of solving OP. Specifically, it has a module that learns to switch attention to another object if it infers that the object contains or carries th target. Our empirical evaluation shows that these components are needed for improving accuracy in this task.

Our results highlight a remaining gap between perfect perception and a pixel-based detector. It is expected that this gap may be even wider when applying OP to more complex natural videos in an open-world setting. It will be interesting to further improve detection architectures in order to reduce this gap.

Acknowledgments

This study was funded by a grant to GC from the Israel Science Foundation (ISF 737/2018), and by an equipment grant to GC and Bar-Ilan University from the Israel Science Foundation (ISF 2332/18). AG received funding from the European Research Council (ERC) under the European Unions Horizon 2020 research and innovation program (grant ERC HOLI 819080).

References

Supplementary Material

Appendix 0.A Appendix 0.A 0.A Appendix 0.A <tag close=" ">Appendix 0.A</tag>Erorr Analysis across the Video Corpus 0.AErorr Analysis across the Video Corpus

Videos in our dataset vary substantially in terms of what OP tasks they involve. This has a large effect over localization accuracy, because it is much harder to localize a carried target than a visible one. To gain more insight into the performance of the leading models, we compare the localization IoU on a video-by-video basis.

Figure depicts per-video IoU of OPNet and two other strong baselines. Each point corresponds to one video and the color reflects the type of frames in that video. Figure (a) shows how OPNet outperforms Transformer on videos including carried frames (colored in orange). Clearly, videos with carried frames are clustered in the lower half of the figure, where OPNet is superior.

Figure S8 Figure S8 S8 Figure S8

(a) (b)

S8Sample-by-samples comparison of OPNet with two strong baselines. Each point represents the IoU of a video from the test set, achieved by OPNet and a baseline. (a) Videos with more than 7% carried frames are colored in orange. (b) Videos with more than 7% occlusion frames are colored in green. Points in the lower part corresponds to videos in which OPNet is superior. Figure S8Sample-by-samples comparison of OPNet with two strong baselines. Each point represents the IoU of a video from the test set, achieved by OPNet and a baseline. (a) Videos with more than 7% carried frames are colored in orange. (b) Videos with more than 7% occlusion frames are colored in green. Points in the lower part corresponds to videos in which OPNet is superior.

Similarly, Figure (b) compares OPNet with the OPNet (LSTM + MLP) baseline, which contains only the first reasoning component (see Our Approach section). It shows that OPNet outperforms the baseline on videos including a high number of occlusion frames (colored in green). It also emphasizes that for most videos, OPNet is superior, as illustrated by the great number of points in the lower half of the figure.

Appendix 0.B Appendix 0.B 0.B Appendix 0.B <tag close=" ">Appendix 0.B</tag>Implementation Details 0.BImplementation Details

We trained OPNet and baseline variants using $L 1$ loss optimized using Adam optimizer with $= β 1 0.9$ , $= β 2 0.999$ , $= ε - ⁢ 1 e 08$ , and using a batch size of 16. We initialized the learning rate to $0.001$ and employed a learning rate decay policy, which reduced the learning rate by a factor of 0.8 every 3 epochs without loss improvement. We tuned all hyperparameters using the validation set. We experimented with using a higher initial learning rate of $- ⁢ 1 e 2$ , but it turned out to be too noisy for the relatively small loss induced by the $L 1$ loss. We also tried lower learning rate ( $- ⁢ 1 e 4$ ), but it did not converge to a good minimum.

The model was trained for 160 epochs, which we verified via manual inspection to be sufficient for convergence of all models. Early stopping was based on the validation-set mean IoU.

For comparisons with CATER [] (Table of the main paper), we used the accuracy values reported in their paper.

For the learning only from visible frames setup (Section and Table of the main paper) we used the values $= α 1$ and $= β 0.5$ . We used these values to normalize the different scales of $L ⁢ l o c a l i z a t i o n$ and $L ⁢ c o n s i s t e n c y$ . We verified via manual inspection that (1) for the first 60-70 epochs the loss component $L ⁢ l o c a l i z a t i o n$ is significantly greater than the loss component $L ⁢ c o n s i s t e n c y$ . Thus, in this phase the model improves its prediction when the target is visible; (2) After 60-70 epochs the two loss components have the same scale. Thus, in this phase the model improves its prediction also when the target is not-visible.

Appendix 0.C Appendix 0.C 0.C Appendix 0.C <tag close=" ">Appendix 0.C</tag>LA-CATER Dataset Preparation 0.CLA-CATER Dataset Preparation

Our new LA-CATER dataset augments the CATER dataset [] with ground-truth locations of all objects and with detailed frame level annotations. Also, instead of using the videos released by CATER we generated new videos using their configuration, and expanded their code to add ground-truth locations and frame-level annotations.

We now describe how we classify frames into the four corresponding OP subtasks. The CATER dataset annotates each frame with the actions occurring for each object in that frame. These actions are defined as follows:

• item 1st item

Slide. Object changes its position by sliding on the XY-plane.

• item 2nd item

Pick-Place. Object is picked up in the air along the Z-axis, moved to a new location and placed down.

• item 3rd item

Contain. A special action performed by cones only, in which cone execute Pick-Place action and positioned on top of another object.

• item 1st item

Contained Frames. We classify a frame as Contained when the target is contained by a cone. Explicitly, a frame is classified as Contained when it is annotated with the “contain” action in CATER, with a cone marked as the containing object and the target marked as the contained object. A frame with recursive containment, namely, a containing cone is itself contained by another cone, is also considered to be a contained frame. Frames are marked as contained from the moment the target is covered and until the containing object is picked up as part of pick-place action.

• item 2nd item

Carried Frames. We mark a frame as Carried when the target is contained by a cone (its action is marked in CATER as contained) and slides along with it. Frames are marked as carried from the beginning of the slide action until the end of the slide action. Thus, only frames corresponding to the slide action are marked as carried.

• item 3rd item

Occluded Frames. For frame $t$ , we define the occlusion rate (OR) of object $x$ by object $y$ as

(S4) Equation S4 S4

= ⁢ O R x t (y) {∩ ⁢ A r e a x t ⁢ A r e a y t ⁢ A r e a x t ≤ ⁢ A r e a x t ⁢ A r e a y t 0 ⁢ O t h e r w i s e

Where $⁢ A r e a x t$ is the area of object $x$ in frame $t$ .

We define the distance from camera (DC) of object $x$

(S5) Equation S5 S5

= ⁢ D C x t ∥ - ⁢ l o c x t ⁢ l o c ⁢ C A t ∥ 2

$⁢ l o c t x$ , $⁢ l o c t ⁢ C A$ denote the 3D coordinates location of object $x$ and the camera in frame $t$ respectively.

We define an indicator for a fully occluded (FO) object:

(S6) Equation S6 S6

= ⁢ F O x t {1 ∃ ⁢ y s . ⁢ t O R x t (y) = ⁢ 1 and D C x t ≥ ⁢ D C y t 0 ⁢ O t h e r w i s e

We then mark frame $t$ as Occluded when the target is fully occluded by another object. e.g $= ⁢ F O ⁢ t a r g e t t 1$

• item 4th item

Visible Frames. Finally, we define frame as Visible when the target is not Contained, Carried or Occluded. Thus, the target needs to be only partially visible to be considered as visible. For instance, the target is still considered visible when it is 20% occluded (e.g $∃ ⁢ y s . = ⁢ t O R x t (y) 0.2$ )

Appendix 0.D Appendix 0.D 0.D Appendix 0.D <tag close=" ">Appendix 0.D</tag>Annotating Frames in Perfect Perception 0.DAnnotating Frames in Perfect Perception

For the perfect-perception setup, we extend the definition of fully occluded (FO) objects from Eq . We define an object to be partially occluded (PO) with respect to the rate $p$ as follows:

(S7) Equation S7 S7

= ⁢ P O x t (p) {1 ∃ ⁢ y s . ⁢ t O R x t (y) ≥ ⁢ p and D C x t ≥ ⁢ D C y t 0 ⁢ O t h e r w i s e

We say that object $x$ is non-visible in frame $t$ with respect to $p$ if $= ⁢ P O x t (p) 1$ . We use the value $= p 0.7$ to decide which objects are non-visible. Contained objects are defined as non-visible, regardless of their $⁢ P O$ value.

Objects are represented by a 5-coordinate vector, containing 4 bounding box coordinates in $(x 1, y 1, x 2, y 2)$ format and an additional visibility bit. Visible objects are represented by their ground-truth bounding boxes and a turned-on visibility bit. Non-visible objects are represented by a four-zeros bounding box coordinates and a turned-off visibility bit.