Acknowledgements

{tcolorbox}

[ colback=abstractBoxBG, colframe=abstractBoxBG, boxrule=0pt, arc=4pt, left=12pt, right=12pt, top=10pt, bottom=12pt, width=enlarge left by=0mm, enlarge right by=0mm ]

Discovering and using Spelke segments Rahul Venkatesh $*, 1, †$ , Klemen Kotar $*, 1$ , Lilian Naing Chen $*, 1$ , Seungwoo Kim $1$ , Luca Thomas Wheeler $1$ , Jared Watrous $1$ , Ashley Xu $1$ , Gia Ancone $1$ ,Wanhee Lee $1$ , Honglin Chen $2$ , Daniel Bear $3$ , Stefan Stojanov $1$ , Daniel Yamins $1, †$ $1$ Stanford University, $2$ OpenAI, $3$ Noetik Inc. Abstract. Segments in computer vision are often defined by semantic considerations and are highly dependent on category-specific conventions. In contrast, developmental psychology suggests that humans perceive the world in terms of Spelke objects—groupings of physical things that reliably move together when acted on by physical forces. Spelke objects thus operate on category-agnostic causal motion relationships which potentially better support tasks like manipulation and planning. In this paper, we first benchmark the Spelke object concept, introducing the SpelkeBench dataset that contains a wide variety of well-defined Spelke segments in natural images. Next, to extract Spelke segments from images algorithmically, we build SpelkeNet, a class of visual world models trained to predict distributions over future motions. SpelkeNet supports estimation of two key concepts for Spelke object discovery: (1) the motion affordance map, identifying regions likely to move under a poke, and (2) the expected-displacement map, capturing how the rest of the scene will move. These concepts are used for “statistical counterfactual probing”, where diverse “virtual pokes” are applied on regions of high motion-affordance, and the resultant expected displacement maps are used to define Spelke segments as statistical aggregates of correlated motion statistics. We find that SpelkeNet outperforms supervised baselines like SegmentAnything (SAM) on SpelkeBench. Finally, we show that the Spelke concept is practically useful for downstream applications, yielding superior performance on the 3DEditBench benchmark for physical object manipulation when used in a variety of off-the-shelf object manipulation models. Project page: https://neuroailab.github.io/spelke_net.

^$†$Corresponding authors: rahulvenkk@gmail.com, dyamins@gmail.com.

*Equal contribution. Author order randomly decided.

Figure 1 Figure 1 1 Figure 1 1 Overview of SpelkeNet’s capabilities. Our model first predicts a probability of motion map, indicating regions likely to undergo movement independent of camera motion—i.e. candidate movable objects. We sample a point from this map and apply a virtual poke. Conditioned on this intervention, our model completes the flow field. From this, we extract a grouping of pixels, or a “segment” corresponding to an entity that would move as a cohesive whole under the application of external forces (i.e. a Spelke object). On the right, we illustrate how these discovered segments can be used in a physical object editing pipeline to precisely define the object we desire to manipulate—ensuring that edits are applied to groups of pixels that would move together in the real world as opposed to segments defined based on appearance or semantics. We show in this paper that the Spelke segments we discover enable more physically plausible object manipulation. Figure 1 Overview of SpelkeNet’s capabilities. Our model first predicts a probability of motion map, indicating regions likely to undergo movement independent of camera motion—i.e. candidate movable objects. We sample a point from this map and apply a virtual poke. Conditioned on this intervention, our model completes the flow field. From this, we extract a grouping of pixels, or a “segment” corresponding to an entity that would move as a cohesive whole under the application of external forces (i.e. a Spelke object). On the right, we illustrate how these discovered segments can be used in a physical object editing pipeline to precisely define the object we desire to manipulate—ensuring that edits are applied to groups of pixels that would move together in the real world as opposed to segments defined based on appearance or semantics. We show in this paper that the Spelke segments we discover enable more physically plausible object manipulation.

Acknowledgements

This work was supported by the following awards: Simons Foundation grant SFI-AN-NC-GB-Culmination-00002986-05, National Science Foundation CAREER grant 1844724, National Science Foundation Grant NCS-FR 2123963, Office of Naval Research grant N00014-20-1-2589, ONR MURI N00014-21-1-2801, ONR MURI N00014-24-1-2748, and ONR MURI N00014-22-1-2740. We also thank Stanford HAI, Stanford Data Science, the Marlowe team, and the Google TPU Research Cloud team for providing computing support.

References