Direct inference of component objects and background from a single image can be difficult, especially when images are complex and when objects are of vastly different sizes. An inference network must learn to pay attention to coarse features in order to segment large objects, and to fine details in the same image order to segment the smaller objects. Inspired by how people solve this problem by stitching together multiple foveations into a coherent whole, we train our models and apply inference on overlapping sub-patches of an image (Figure ).

In particular, given an image of size $\times HW$, we divide the image into a $\times 88$ grid (pictured in the left of Figure ), with each grid element having size $/\times /H8W8$. We construct a sub-patch for every $\times 22$ component sub-grid, leading to a total of 64 different overlapping sub-patches. We apply inference on each sub-patch. Under this decomposition, smaller objects still appear large in each sub-patch, while larger objects are shared across sub-patch.

To obtain a global segmentation map, we merge each sub-patch sequentially using a sliding window (Figure ). At each step, we iterate through each segment given by the inference model from a sub-patch, and merge it with segments obtained from previous sub-patches, if there is an overlap in masks above 20 pixels. Every segment that does not get merged is initialized as a new object.

Our backprojection model maps a mask $mk$ to an underlying 3D primitive cuboid, represented as a translation $\in tkR3$, size $\in skR3$, and rotation $\in qkR3$ (as a Euler angle) transform on a unit cuboid in a fully differentiable manner. In our case, we primarily pre-train a neural network to approximate the 2D-to-3D projection and use it as our differentiable backprojection model. However we show that such a task can also be approximated by assuming the camera parameters and the height of the plane is given (ignoring size and rotation regression), with little reduction in performance (see Appendix  for further details).

We construct a dynamics model over the next state of different physical objects $(tkt,skt,qkt,mkt)$ by using first order approximation of velocity/angular velocity of the states of the object. Specifically, our model predicts

Our $Render(\cdot )$ function computes the segmentation mask of a predicted physical object, assuming all other physical objects are rendered. To compute this, we initialize a palette $=p01$, which we define to represent the context in an image that has not been rendered yet. We further utilize a separate pre-trained model Project$(\cdot )$ that projects each primitive in 3D to a 2D segmentation mask (the inverse of the backprojection model described above). We then reorder predicted physical objects in increasing distance from the camera to $(^tk\prime t,^sk\prime t,^qk\prime t)$. We then sequentially render each predicted physical object using $=Render(^tk\prime t,^sk\prime t,^qk\prime t)p-k\prime 1Project(^tk\prime t,^sk\prime t,^qk\prime t)$, and update the corresponding palette to be rendered as $=pk\prime p-k\prime 1(-1Project(^tk\prime t,^sk\prime t,^qk\prime t))$.

Given modeled future states, the overall likelihood of a physical object $(tkt,skt,qkt,mkt)$ is given by

where we assume Gaussian distributions over translation, sizes, and rotations with $\sigma s=\sigma t=\sigma q=1$. $p(\cdot )$ is the probability of a predicted mask, defined as $=p(^mkt,mkt)+1>mkt0.5^mkt(-11>mkt0.5)(-1^mkt)$, where $1(\cdot )$ is the indicator function on each individual pixel. Overall, our dynamics model seeks to enforce that objects have zero order motion and maintain shape.

We represent images $\in xtRD$ at each time step as spatial Gaussian mixture models, with each mixture model being defined by a segmentation mask $m$ in $M$ (Section ). Each corresponding latent $zk$ is decoded to a pixel-wise mean $\mu ik$ and a pixel-wise mask prediction $dik$ using a VAE decoder $Decode(\mu k,|dkzk)$. We assume each pixel $i$ is independent conditioned on $z$, so that the likelihood becomes

for background component $mb$, $\mu b$, $db$ and object components $mk$, $\mu k$, $dk$, where $p\theta (dik|zk)=p\theta (dik=mik|zk)$, is the probability that decoded mask from the latent matches the ground truth mask for the mixture. We use $=\sigma 0.11$ and $=\sigma b0.07$ to break symmetry between object and background components, encouraging the background to model the more uniform image components ( , ). Our overall loss encourages the decomposition of an image into a set of reusable sub-components, as well as a large background.

To train models on moving ShapeNet objects, we use the generation code provided in the ADEPT dataset in ( ) . We generate a training set of 1,000 videos, each 100 frames long, of objects (80% of the objects from 44 ShapeNet categories) as well as rectangular occluders. Objects move in either a straight line, back and forth, or rotate, but do not collide with each other.

Videos have a resolution of 1024$\times$1024 pixels. We apply our model with a patch size of 256$\times$256. We use a residual architecture ( , ) for the attention and VAE components. Our backprojection model is pretrained on scenes of a single ShapeNet object, varied across different locations on a plane, with different rotations, translations, and scales. Our backprojection model only serves as a rough relative map from 2D mask to corresponding 3D position/size, as the dataset they are trained on utilize separate camera extrinsics/intrinsics than the ADEPT dataset, and also do not exhibit occlusions. To compute the physical plausibility $Lphysics$ of primitives, we utilize the observations from the last three time steps. For efficiency, we evaluate physical plausibility on each component sub-patch of image. We train a recurrent model with a total of 5 slots for each image. Image segmentation is trained and evaluated on a per frame basis.

To quantify our results, we measure the intersection over union (IoU) between the predicted segmentation masks and the corresponding ground truth masks. We compute the IoU for each ground truth mask by finding the maximum IoU intersection with a predicted mask. We report the average IoU across all objects in an image, as well as the percentage of objects detected in an image (with IoU $>0.5$). To measure 3D inference ability, we report the maximum 3D IoU intersection between each ground truth 3D box and our inferred 3D bounding box. We also report the recall of ground truth 3D objects ( , ) detected in an image (with 3D IoU threshold $>0.1$). We utilize our backprojection model to extract 3D bounding box proposals from 2D segmentations and apply a linear transformation to align coordinate spaces.

We compare with two recent models of self-supervised object discovery, OP3 ( , ) and MONet ( , ), as well as three algorithms for object segmentation, Normalized Cuts ( , ), Crisp Boundary  ( , ), and the recent UVOD ( , ). We train OP3 with 7 slots, with 4 steps of optimization per mask in the first image, and an additional step of optimization per future timestep. Due to memory constraints, we were only able to train the OP3 model on inputs of size 128 by 128. We train MONet on inputs of size 256 by 256. We apply normalized cuts on a region adjacency graph of the 256 by 256 image, and train UVOD in 256 by 256 images . We also compare with ablations of POD-Net: POD-Net applied directly to an image (single-scale) as opposed to across patches (multi-scale), POD-Net without physics, and POD-Net with a hard-coded backprojection model (‘Manual’) .

We quantitatively compare object masks discovered by our model and other baselines in Table . We find that OP3 performs poorly, as it only discovers a limited subset of objects. MONet performs better and is able to discover a single foreground mask of all objects. However, the masks are not decomposed into separate component objects in a scene (Figure , 2nd row). Our scenes consist of a variable set of objects of vastly different scales, making it hard for MONet to learn to assign individual slots for each object. We find that a baseline based on normalizing cuts/crisp boundary detection is also able to segment objects, but is unable to get sharp segmentation boundaries for each object, and often decomposes a single object into multiple subobjects (see Appendix  for details). Finally, UVOD also only segments a single foreground object.

We find that applying POD-Net (single scale, no physics) improves on MONet slightly, discovering several different masks containing multiple objects, albeit sometime missing other objects. POD-Net (single scale, physics) more reliably segments separate objects, but still misses objects. POD-Net (multi scale, no physics) reliably segments all objects in the scene, but often merges multiple objects into one object, especially when objects are overlapping (e.g., Figure , 3rd row). Finally, POD-Net obtains the best performance and segments all objects in the scene and individual objects where multiple objects overlap with each other (Figure , 4th row). Utilizing a manually coded backprojection module, POD-Net (‘Manual’) only leads to slight degradation in performance.

Next we analyze the 3D objects discovered by POD-Net. In Table , POD-Net performs the best, achieving the highest 3D average IoU intersection and recall. Crisp boundary detection obtains a high average IoU but low recall due to a large number of proposals. All IoUs are low due to the challenging nature of the task – obtaining high 3D IoU requires correct regression of size, position, and depth (using only RGB inputs). Even recent supervised 3D reconstruction approaches using 0.25 IoU thresholds for evaluation ( , ). Visualizations of the discovered objects in Figure  show that POD-Net is able to segment a scene into a set of temporally consistent 3D cuboid primitives. We further find a high correlation $=r0.615$ between the predicted and ground truth translation of objects and show plots of correlations in Figure  of ground truth and predicted objects.

Just as young children can detect and reason about new objects with arbitrary shapes and colors, we test how well POD-Net can generalize to scenes with both novel objects and colors. We evaluate the generalization of our model on two datasets: a novel object dataset consisting of 20 new objects and a novel color dataset, where each object is split into two colors.

Figure  shows a quantitative analysis of POD-Net applied to both novel objects and colors. We find that in both settings, POD-Net with physical consistency gets better segmentation than without. Performance is higher here compared to that reported on the training set, as both novel datasets contain fewer objects in a single scene. Qualitatively, POD-Net performs well when asked to discover novel objects, although it can mistake a multicolored novel shape to be two objects.

We use the dataset in ( ) with 492 videos of real block towers, which may or may not be falling. Each frame contains 2 to 4 blocks of red, yellow, or green color. Each block has the same 3D shape, although the 2D projections on the camera differ.

For our backprojection model, we use a pretrained neural network on scenes of a single block at different heights, sizes, and varying distances. Similar to Section , the backprojection model serves as a rough 2D to 3D model and is trained with different camera and perspective parameters without occlusion. All other settings are the same as in Section .

We compare masks discovered by POD-Net and baselines in Figure . We find that OP3 and MONet often misses blocks and also groups two blocks into a single object, leading to floating blocks in the air (Figure , 2nd row). Normalized cuts also suffers from a similar issue of grouping blocks, but suffers an additional issue of oversegmentation (see Appendix ). UVOD fails to predict a segmentation due to limited motion in video. POD-Net (single scale, no physics) is able to segment all blocks, but treats the entire stack as a single object. POD-Net (multi-scale, no physics) does better and is able to reliably segment all blocks, though it still groups blocks of similar colors together (Figure , 3rd row). Finally, POD-Net with multiple scales and physical consistency performs the best, reliably separating individual blocks in a tower (Figure , 4th row).

We use POD-Net trained in Section  to obtain a set of physical objects (represented as cuboids) describing an underlying scene. The extracted objects are provided as a scene description to the stochastic physics engine and particle filter described in  ( ) . We evaluate our models using a relative accuracy metric ( , ).

On the Block task, we find that our model achieves a relative accuracy of 0.622. Its performance on a single video can be seen in Figure , where it has learned to localize the block well enough that the model is surprised when it appears on the other side of the wall. The model in ( ) scores a relative accuracy of 0.680; this acts as an upper bound for the performance of our model, since supervised training is used to discover the object masks and recover object properties. In contrast, POD-Net discovers 3D objects in an unsupervised manner, outperforming the baseline generative models studied by ( ) that do not encode biases for objecthood (GAN: 0.44, Encoder-Decoder: 0.52, LSTM: 0.44). On the Overturn (Long) task – the one task where the ADEPT model underperforms baselines – our model obtains a performance of 0.77, outperforming ( ) (0.73), and equalling or exceeding models that do not encode biases for objects (GAN: 0.81, Encoder-Decoder: 0.61, LSTM: 0.63).

A limitation of our approach towards discovering 3D object primitives is that across a long video (over 100 timesteps), there may be several spurious extraneous objects discovered. The model in ( ) does not deal well with such spurious detections either, requiring us to tune separate hyper-parameters for each task.