Using Shape to Categorize: Low-Shot Learning with an Explicit Shape Bias

Using Shape to Categorize: Low-Shot Learning with an Explicit Shape Bias Stefan Stojanov Anh Thai James M. RehgGeorgia Institute of Technology{sstojanov, athai6, rehg}@gatech.edu

It is widely accepted that reasoning about object shape is important for object recognition. However, the most powerful object recognition methods today do not explicitly make use of object shape during learning. In this work, motivated by recent developments in low-shot learning, findings in developmental psychology, and the increased use of synthetic data in computer vision research, we investigate how reasoning about 3D shape can be used to improve low-shot learning methods’ generalization performance. We propose a new way to improve existing low-shot learning approaches by learning a discriminative embedding space using 3D object shape, and using this embedding by learning how to map images into it. Our new approach improves the performance of image-only low-shot learning approaches on multiple datasets. We also introduce Toys4K, a 3D object dataset with the largest number of object categories currently available, which supports low-shot learning. 1 footnote 1 1 footnote 1 The code and data for this paper are available at our project page https://rehg-lab.github.io/publication-pages/lowshot-shapebias/

\SetArgSty

textnormal \SetKwInputKwDefDefine \newcitesNewAppendix References

1 section 1 1 §1 <tag close=" ">1</tag>Introduction

Understanding the role of 3D object shape in categorizing objects from images is a classical topic in computer vision [], and the early history of object recognition was dominated by considerations of object shape. For example, David Marr’s influential theory [] posits that image-based recognition should be formulated as a sequence of information extraction steps culminating in a 3D representation to be used for recognition. The difficulty of reliably extracting 3D shape from images, combined with the availability of large-scale image datasets [], motivated the modern development of purely appearance-based approaches to recognition and categorization. This has culminated in current approaches such as CNNs that learn feature representations directly from images. Moreover, a study by Geirhos et al. [] of the inductive biases of CNNs trained on ImageNet suggests that categorization performance is driven primarily by a bias towards image texture rather than object shape. 2 footnote 2 2 footnote 2 This study does not speak to the possibility of whether shape could be used more effectively, and it is unclear how much of the bias stems from the composition of the ImageNet dataset itself.

However, studies of infant learning [] suggest that shape does play a significant role in the ability to rapidly learn object categories from a small number of examples, a task which is analogous to few-shot learning. Both young children and adults who are forced to categorize novel objects based on a few examples display a shape bias, meaning that shape cues seem to play a dominant role in comparison to color and texture when inferring category membership. These studies beg the question of whether information about 3D object shape could be useful in learning to perform few-shot categorization from images. While prior work has demonstrated effective approaches to object categorization using 3D shapes as input [], and there is a large literature on few-shot learning from images alone [], the question of how shape cues could be used to learn effective representations for image-based low-shot categorization has not been investigated previously.

Figure 1 Figure 1 1 Figure 1 1To perform low-shot learning with shape-bias, we train a embedding space defined by a point cloud encoder

f p

trained with cross-entropy. Shape based embedding spaces are more discriminative than image-based ones (see Tbl. ). We extract object shape embeddings, and train an image encoder

f i

to map images into the shape space. If trained successfully,

f i

will have the discriminative properties of

f p

. Figure 1To perform low-shot learning with shape-bias, we train a embedding space defined by a point cloud encoder

f p

trained with cross-entropy. Shape based embedding spaces are more discriminative than image-based ones (see Tbl. ). We extract object shape embeddings, and train an image encoder

f i

to map images into the shape space. If trained successfully,

f i

will have the discriminative properties of

f p

The goal of this paper is to explore the incorporation of a shape bias in SOTA approaches to few-shot object categorization and thereby investigate the utility of shape information in category learning. We leverage the recent availability of datasets of 3D object models with category labels, such as ModelNet40 [] and ShapeNet []. By sampling surface point clouds and rendering images of these models, we can construct datasets that combine 3D shape and image cues. Unfortunately, however, ShapeNet and ModelNet contain a relatively small number of object categories (55 and 40 respectively), making it difficult to test categorization at a sufficient scale. To resolve this limitation, we introduce a new 3D object dataset, Toys4K consisting of 4,179 3D objects from 105 object categories, designed to contain categories of objects that are commonly encountered by infants and children during their development.

We report on two sets of investigations. First, we examine the relative effectiveness of purely image-based and purely shape-based approaches to few-shot categorization. We demonstrate that purely shape-based few-shot learning outperforms image-based approaches, and establish an empirical upper bound on the effectiveness of a shape bias. Second, we develop a novel approach for training an image embedding representation for low-shot categorization which incorporates an explicit shape bias, which we outline in Figure . We benchmark this approach on a representative set of SOTA few-shot learning architectures and demonstrate that the incorporation of shape bias results in increased generalization accuracy over image-based training alone. In summary, this paper makes the following contributions:

• item 1st item

A new approach to add explicit shape-bias to existing low-shot image classification methods, utilizing 3D object shape to learn similarity relationships between objects, which leads to improved low-shot performance.

• item 2nd item

The first evidence that shape information can enable image-based low-shot classifiers to generalize with higher accuracy to novel object categories.

• item 3rd item

Toys4K - new 3D object dataset containing approximately twice the number of object categories as previous datasets which can be used for low-shot learning.

2 section 2 2 §2 <tag close=" ">2</tag>Related Work

Object Recognition from Synthetic Data

A large body of work focuses on appearance [], point cloud [] and voxel [] based recognition of synthetic object data with category taxonomies based on object shape such as ModelNet40 []. The trade-offs between learning using point clouds, depth maps, voxels, or images have been studied by [] but their study focuses on standard supervised classification and does not extend to low-shot classification of novel object categories or on combining shape and appearance information during learning.

Low-Shot LearningLow-shot learning algorithms can be categorized into two broad sets. Optimization-based algorithms such as MAML [], LEO [], and Reptile [], which during the base-classes training stage, attempt to learn a representation that can quickly be adapted using small amounts of information with gradient-based learning in the low-shot learning stage. Metric learning-based methods such as Prototypical [], Matching [], and Relation [] networks, as well as the more recent SimpleShot [], FEAT [], and RFS [] aim to use the base class data to learn a similarity metric that will also be discriminative for novel classes during the low-shot phase. Despite their simplicity, metric-based approaches have superior performance on low-shot learning benchmarks []. Our approach of adding shape bias belongs to the latter category, and compared to both is the first approach to combine both appearance and shape information for low-shot learning.

3D Object Shape Datasets Other related works focus on building datasets of 3D object models for recognition, single image object shape reconstruction and shape segmentation []. The most widely used 3D shape datasets with category labels are ModelNet40 [] with 12K object instances of 40 categories with no object surface material properties, ShapeNetCore.v2 [] with 52K objects of 55 object categories with basic surface texture properties (basic shading and UV mapping, but no physically based materials). The ShapeNetSem split of ShapeNet consists of over 100 categories but is unsuitable for recognition since individual object instances are assigned to multiple categories. Datasets such as ABC [] and Thingi10k [] claim higher mesh quality than previous datasets but lack object category annotation, making them more suitable for low-level tasks like surface normal estimation and category agnostic shape reconstruction. The ModelNet40 and ShapeNet datasets were scraped from online repositories and have categories largely based on the data that was available in these repositories. In contrast, our new Toys4K is curated specifically for testing the generalization ability of learned representations to new classes. Compared to the aforementioned datasets, Toys4K consists of highly diverse object instances within a category (evident in Figure , detailed composition is included in the supplement) and has the highest number of individual object categories despite its smaller total size.

Multi-modal LearningAligning representations from different data modalities has been extensively studied in vision and language works on zero-shot learning []. More recently, Schwartz et al. [] and Xing et al. [] improve low shot image classification performance on standard low-shot datasets by combining the representation learned through the appearance modality (images) with language model word vector embeddings. In comparison, we combine appearance (images) and shape (point clouds) to learn a representation for low shot object recognition that is biased to object shape and leads to better low-shot generalization. It is important to note that these works use multi-modal information for the low-shot queries at test time, whereas our approach only uses multi-modal information for the low-shot support set.

Another category of multi-modal learning works focuses on learning joint embedding spaces of 3D meshes and images for image-based 3D shape retrieval []. While these works focus on retrieval for the same object categories at training and testing time, our work focuses on combining appearance and shape information for low-shot generalization to novel object categories.

3 section 3 3 §3 <tag close=" ">3</tag>Using Shape for Low-Shot Classification

In principle, 3D shape is an attractive representation for object recognition [] due to its invariance to the effects of viewpoint, illumination, and background, which can be challenging for appearance-based approaches. While appearance-based methods may be able to model these sources of variation given sufficient training images, there is always a question of how well such models can generalize to novel categories and objects [].

Despite its potential advantages, no previous work on low-shot learning has utilized 3D shape, for at least two reasons: 1) It is unclear how to leverage 3D shape in improving image-based low shot learning; 3 footnote 3 3 footnote 3 Our focus is on few-shot methods in which the queries are images, with no 3D shape information available, as this is the most general and useful paradigm. 2) There is a lack of 3D shape datasets that contain a sufficient number of object categories to support effective experimentation. This is due to the additional data requirements of few-shot learning: The training/validation/testing split is over different classes and not data points of the same class [] in order to effectively test generalization to unseen classes.

To explain this issue more formally, let $D train$ denote the base classes, and $D val$ and $D test$ denote the validation and testing sets, respectively, where these sets comprise a disjoint partition of the total available classes. The base classes must be sufficiently large and diverse to learn an effective feature representation in the training phase, and the $D val$ set must similarly support the accurate assessment of low-shot generalization ability during hyperparameter tuning (i.e. model selection while training on the base classes). The $D test$ set is used to generate labeled low-shot training examples (supports), and unlabelled low-shot testing examples (queries), which are used to evaluate the generalization performance of the model at testing time, which we refer to as the low-shot phase. As a result of these constraints, the standard 3D shape datasets ModelNet40 [] and ShapeNet55 [] can only support 10-way and 20-way testing, respectively. If the number of testing classes is insufficient, the estimation of the generalization performance of the method may be inaccurate.

In this section, we describe our two primary contributions which address the limitations described above. In § we present our novel method for introducing shape bias in learning a low-shot image representation. In § , we introduce a novel 3D object category dataset, Toys4K, consisting of 4,179 object instances organized into 105 categories, with an average of 35 objects (3D meshes) per category. Toys4K supports up to 50-way classification, expanding well beyond ModelNet40 and ShapeNet55 (see Fig. ).

Figure 2 Figure 2 2 Figure 2

2(a) The standard setting: Prototypes are formed from images. (b) Our novel shape-biased setting: Image and shape embeddings are averaged. In both cases, the image-only queries

q i

can be classified by identifying the closest prototype

p j

. The training process for the mapping functions

f i

and

f p

is illustrated in Figure . Figure 2(a) The standard setting: Prototypes are formed from images. (b) Our novel shape-biased setting: Image and shape embeddings are averaged. In both cases, the image-only queries

q i

can be classified by identifying the closest prototype

p j

. The training process for the mapping functions

f i

and

f p

is illustrated in Figure . Table 1 Table 1 1 Table 1

1On ModelNet40-LS, low-shot generalization is higher for point-cloud based learning than image based learning, justifying our approach in combining the modalities. Oracle model has access to both image and point cloud information. See text for details.

Table 1On ModelNet40-LS, low-shot generalization is higher for point-cloud based learning than image based learning, justifying our approach in combining the modalities. Oracle model has access to both image and point cloud information. See text for details.
	5-way		10-way
	1-shot	5-shot	1-shot	5-shot
Image	58.99	74.29	45.82	62.73
Point Cloud	66.02	83.61	54.44	75.26

Img + Ptcld
Oracle
68.04	82.07	57.03	73.11

3.1 subsection 3.1 3.1 §3.1 <tag close=" ">3.1</tag>Low-Shot Learning with Shape Bias

We begin by describing the problem formulation: We assume that shape data in the form of 3D point clouds is available for each RGB image in a dataset. We achieve this by rendering RGB images from the 3D models. 3D shape information is used directly during training and validation, in order to construct a representation with an explicit shape bias. In addition, during the low-shot phase, episodes are generated so that point clouds are available for the support objects, but not for the query objects. This assumption allows for both appearance and shape information to be used in building class prototypes, but inference is done using images only. The distinction between image only low-shot learning and our new setting is illustrated in Figure .

In this work, we adopt a low-shot learning approach based on a metric embedding space. In this approach, $D train$ is used to learn a function $f i$ that maps the input data into an embedding space where object instances of the same category are close and instances of different categories are far apart, according to some distance metric. This mapping can be fixed after being learned from $D train$ or fine-tuned further, depending upon the algorithm design. During the low-shot phase, the supports and queries are mapped into the embedding space (see Figure ), and the queries are classified according to a nearest neighbor or nearest class prototype (e.g. support centroid) rule. Metric-based low-shot learning has high accuracy [] and is significantly more computationally efficient than approaches that fine-tune on the low-shot supports. We first demonstrate that shape-based low-shot learning allows for better generalization than image-based low-shot learning, and then show how a shape-based embedding with high generalization ability can be used to improve image-based low-shot classification.

Shape-based low-shot learning outperforms image-based low-shot learning

We perform a simple empirical study to determine whether shape has an advantage for low-shot generalization. We train two embedding spaces, one using image data and one using point cloud data. For each type of data, we follow the SimpleShot [] approach, meaning that we train a classifier using cross-entropy on $D train$ and use the learned feature space (output of the last pooling layer) to perform nearest centroid-based low-shot classification in normalized Euclidean space. We use a ResNet18 [] for image learning and a DGCNN [] for point cloud learning on the ModelNet40-LS dataset (see § ).

We present the results in Tbl. , and as might be expected, see significantly higher low-shot performance for the point cloud model relative to the image model. This quantifies the improvement in generalization to novel categories as as result of using a 3D shape-based representation and suggests that 3D shape can yield a more discriminative embedding space. The question then is how can this benefit be retained when testing the model on image data alone?

Combining Appearance and Shape

Figure illustrates our approach to using the 3D shape information available at training time in order to learn how to embed the image-only queries. First, we train a low-shot point-cloud based classifier on the set of base-classes $D train$ , resulting in an a highly discriminative embedding space for both seen and novel categories. We then extract point cloud embeddings for each object in the training set and train a CNN to map images into the shape embedding space.

Let $D$ be a dataset of paired object point clouds $o p$ and images $o i$ , partitioned into $D train$ , $D val$ , and $D test$ . Let $: ⁢ f p (x) → × N R 3 R d$ denote the trained function for mapping point clouds of size $N$ into an embedding space of dimension $d$ . This embedding space is optimized to yield favorable metric properties for low shot classification, using the labelled point cloud data in $D train$ . Our goal is then to learn a second mapping, $: ⁢ f i (x) → R × H W 3 R d$ , where $H, W$ are the image height and width, from images into the shape embedding space defined by $⁢ f p (x)$ . We denote point cloud embeddings as $= ⁢ f p (o p) ϕ p$ and image embeddings as $= ⁢ f i (o i) ϕ i$ .

Figure 4 Figure 4 4 Figure 4

3Approximately one third of the objects in Toys4K, a new dataset of 3D assets for low-shot object learning using object appearance and shape information. Figure 3Approximately one third of the objects in Toys4K, a new dataset of 3D assets for low-shot object learning using object appearance and shape information.

4The high number of categories in Toys makes low-shot learning on Toys4K a challenging task. Figure 4The high number of categories in Toys makes low-shot learning on Toys4K a challenging task. Table 2 Table 2 2 Table 2

2Toys4K has the most categories of any available dataset of 3D objects.

Table 2Toys4K has the most categories of any available dataset of 3D objects.
Dataset	Instances	Categories
Toys4K	4,179	105
ModelNet40 []	12,311	40
ShapeNet []	52,000	55
Thingi10K []	10,000	N/A
ABC []	750K	N/A

We train a model that learns the mapping from images to shape embeddings by minimizing two loss functions (see part 3 of Figure ). For a mini-batch $⊂ B D train$ the first loss minimizes the squared Euclidean distance (which we denote as $⁢ d (x, y)$ ) between the learned point cloud embeddings, and the image based embeddings

= L 1 ∑ ∈ (o i, o p) B ⁢ d (ϕ i, ϕ p) .

The second loss constrains the pairwise distances between the image embeddings of different object instances to be the same as the pairwise distances of the learned shape embeddings. Let $I$ denote the set of all $= (k, l) ((o i k, o p k), (o i l, o p l))$ object instance data pairs in a mini-batch. We define the second loss as

= L 2 ∑ ∈ (k, l) I (- ⁢ d (ϕ p k, ϕ p l) ⁢ d (ϕ i k, ϕ i l)) 2 .

During training, both losses are minimized with equal weight. Validation for choosing $f i$ is done by nearest centroid classification on $D val$ . In Section we show that minimizing only $L 1$ results in convergence without learning to match the distribution of the shape embedding well on the training set, resulting in poor performance.

Inference: During the low-shot phase, as shown in Figure , class prototypes are built by averaging the shape $ϕ p$ and image $ϕ i$ embeddings for each support object, whereas only image information is used to map the query objects via $f i$ . The queries are classified based on the nearest centroid to the query embedding. This inference procedure is used for all algorithms in this paper that combine both image and shape information, with the exception of FEAT [], which uses an additional set-to-set mapping.

It is important to understand how the shape-biased encoder performs when there is no explicit shape information available in the low-shot phase, and what is the gain in accuracy by making shape available for building class prototypes. To this end, in § we also evaluate the setting where there are no point clouds available in the low-shot phase.

Why is mapping images to shape embeddings difficult? If the mapping $⁢ f i (x)$ is learned exactly, it would map images to their corresponding point cloud embeddings so that

∈ ∀ (o i, o p) {∪ D train D val D test}, = || - ϕ i ϕ p || 2 0 .

This is challenging, however, since $f i$ can only be trained on the base classes in $D train$ , requiring it to correctly extrapolate to the metric properties of objects from novel classes.

We perform a simple test to validate the feasibility of mapping images to shape embeddings in general and establish an empirical upper bound. We perform this by simply minimizing the $L 2$ distance between the images and their corresponding shape embeddings on combined data from base classes, validation and test classes ( $= D train {∪ D train D val D test}$ ). This model is referred to as Image + Point Cloud Oracle in Table and provides empirical evidence that it is possible to learn how to map images into a shape embedding space with high accuracy when all of the data is available. This model’s performance closely matches that of the shape-only model, and significantly outperforms the image-based approach, providing further evidence that extrapolating the metric properties of the shape-embedding space to novel categories is the key challenge in learning to map images to shape embeddings.

3.2 subsection 3.2 3.2 §3.2 <tag close=" ">3.2</tag>Toys4K Dataset

An object dataset with a high number of diverse categories and high-quality 3D meshes is essential to study whether leveraging 3D object shape can enable improved low-shot generalization. We satisfy this requirement with our new Toys4K dataset. While it is possible to use existing datasets such as ModelNet40 and ShapeNet (which we include in our experiments), the limited number of categories is an obstacle to few-shot learning. For example, applying standard training/validation/test ratios (e.g. from mini-ImageNet []) to the 40 categories in ModelNet40 results in a 20-10-10 split, which limits the possibilities for many-way testing. A comparison of Toys4K to prior datasets is available in Table . In Figure we demonstrate that many-way low-shot classification on Toys4K is a challenging task in comparison to ModelNet40 and ShapeNet.

Toys4K consists of 4,179 object instances in 105 categories, with an average of 35 object instances per category with no less than 15 instances per category, allowing for 5 support 10 query low-shot episodes to be formed. Fig. provides an example of the quality and variety of the models. Further details on the dataset composition are available in the supplement. Toys4K was collected by selecting freely-available objects from Blendswap [], Sketchfab [], Poly [] and Turbosquid [] under Creative Commons and royalty-free licenses. Our list of object categories was developed in collaboration with experts in developmental psychology to include categories of objects available and relevant to children in their infancy. We manually selected each object and manually aligned the objects within each category to a canonical coordinate system that is consistent across all instances in that category.

4 section 4 4 §4 <tag close=" ">4</tag>Experiments

In this section, we perform an empirical evaluation of the benefit of explicit shape bias on multiple datasets and image-only low-shot learning algorithms.

4.1 subsection 4.1 4.1 §4.1 <tag close=" ">4.1</tag>Datasets

In addition to our new dataset, Toys4K, we use the 3D object category datasets ModelNet40 [], ShapeNet []. For descriptions of the datasets please refer to § . We render images using the Cycles ray tracing renderer in Blender [] using uniform lighting on white backgrounds. For all datasets, camera pose is randomly sampled for 25 views of each object with azimuth $∈ ψ [0, 360]$ and elevation $∈ θ [- 50, 50]$ degrees. Object surface point clouds are sampled from the 3D object meshes.

Toys4K is our new low-shot learning dataset is described in detail in § . We use a split of 40, 10, 55 for base, low-shot validation, and testing classes, respectively. For Toys and all other datasets, the split is designed such that the categories with most classes are in the training set, and the validation and testing classes are randomly chosen from the remainder of the data.

ModelNet40-LS is the existing ModelNet40 [] dataset, with a 20, 10, 10 split for base, low-shot validation and testing classes respectively.

ShapeNet-LS is the existing ShapeNetCore.v2 [] dataset, with a 25, 10, 20 split for base, low-shot validation and testing classes respectively, using a reduced subset of object samples per category to reduce training time due to the high data imbalance.

4.2 subsection 4.2 4.2 §4.2 <tag close=" ">4.2</tag>Baselines

Regarding low-shot learning, we compare with the classical low-shot learning method Prototypical Networks [], and the state-of-the-art algorithms FEAT [], RFS [], and SimpleShot []. With respect to learning joint embeddings, we compare with a simple triplet loss-based approach that learns joint embeddings of images and point clouds. All baselines use a standard ResNet18 [] as a backbone for image encoding and a DGCNN [] to encode point clouds. In the supplement we perform an ablation study over different point cloud architectures including PointNet [] and PointNet++ []. Our low-shot learning baseline implementations were all validated by re-creating the results from the original papers. 4 footnote 4 4 footnote 4 Experiment implementation details included in the supplement

SimpleShot [] is a simple low-shot learning baseline algorithm that outperforms many recent methods. It makes use of an embedding space learned by a CNN by training on the base training classes for a standard classification task using cross-entropy loss. Validation and testing are done using a nearest neighbor classifier in the learned embedding space, with feature normalization and training set mean subtraction resulting in improved performance.

RFS [] is another simple low-shot learning algorithm that is competitive with many recent approaches. Training the embedding space is done using cross-entropy on the training set, but at testing time, a simple logistic regression classifier is learned for each low-shot episode. In the original work, the authors show that training a set of embedding models with distillation slightly improves performance. We omit this for a fair comparison with all metric-based works since this addition would likely lead to performance improvements across the board.

Table 3 Table 3 3 Table 3

3Results on image-only and shape-biased low-shot recognition on ModelNet40-LS. Parenthesis show confidence intervals based on 5K low shot episodes. Bold indicates best performance between a low-shot learning approach with and without shape bias; underline indicates best overall. Adding shape bias improves performance in the 1-shot learning setting and has competitive performance otherwise.

Table 3Results on image-only and shape-biased low-shot recognition on ModelNet40-LS. Parenthesis show confidence intervals based on 5K low shot episodes. Bold indicates best performance between a low-shot learning approach with and without shape bias; underline indicates best overall. Adding shape bias improves performance in the 1-shot learning setting and has competitive performance otherwise.
Episode Setup $→$	1-shot 5-way	5-shot 5-way	1-shot 10-way	5-shot 10-way
RFS []	56.67 $± 0.30$	72.64 $± 0.26$	43.79 $± 0.16$	60.61 $± 0.11$
ProtoNet []	50.11 $± 0.31$	64.44 $± 0.24$	36.44 $± 0.17$	46.70 $± 0.26$
Triplet	52.53 $± 0.66$	63.07 $± 0.59$	37.24 $± 0.37$	49.79 $± 0.26$
SimpleShot []	58.99 $± 0.29$	74.29 $± 0.24$	45.82 $± 0.17$	62.73 $± 0.11$
Shape Bias (w/ pc) - SimpleShot - $L 1$ only	59.81 $± 0.31$	71.61 $± 0.26$	47.89 $± 0.15$	59.48 $± 0.11$
Shape Bias (w/o pc) - SimpleShot	60.23 $± 0.30$	75.59 $± 0.24$	47.92 $± 0.15$	64.88 $± 0.11$
Shape Bias (w/ pc) - SimpleShot	61.91 $± 0.31$	75.39 $± 0.24$	49.84 $± 0.16$	64.21 $± 0.11$
FEAT []	58.30 $± 0.29$	71.54 $± 0.23$	45.41 $± 0.16$	60.44 $± 0.11$
Shape Bias (w/o pc) - FEAT	60.19 $± 0.31$	74.66 $± 0.25$	48.6 $± 0.16$	64.08 $± 0.11$
Shape Bias (w pc) - FEAT	62.84 $± 0.30$	74.84 $± 0.24$	51.49 $± 0.15$	63.80 $± 0.11$

Table 4 Table 4 4 Table 4

4Results on image-only and shape-biased low-shot recognition on Toys4K. Parenthesis show 95% confidence intervals based on 5K low shot episodes. Bold indicates best performance for a low-shot approach with and without shape bias; underline indicates best overall. Adding shape-bias improves 1-shot performance when the number of low-shot ways is higher.

Table 4Results on image-only and shape-biased low-shot recognition on Toys4K. Parenthesis show 95% confidence intervals based on 5K low shot episodes. Bold indicates best performance for a low-shot approach with and without shape bias; underline indicates best overall. Adding shape-bias improves 1-shot performance when the number of low-shot ways is higher.
Episode Setup $→$	1-shot 5-way	5-shot 5-way	1-shot 10-way	5-shot 10-way	1-shot 20-way	5-shot 20-way	1-shot 30-way	5-shot 30-way
RFS []	67.10 $± 0.71$	81.76 $± 0.54$	52.94 $± 0.51$	71.30 $± 0.45$	40.97 $± 0.32$	59.53 $± 0.30$	34.34 $± 0.26$	53.46 $± 0.24$
ProtoNet []	62.48 $± 0.34$	79.69 $± 0.25$	48.27 $± 0.24$	68.03 $± 0.21$	36.38 $± 0.15$	56.25 $± 0.15$	30.62 $± 0.11$	49.58 $± 0.11$
Triplet	63.87 $± 0.34$	73.95 $± 0.62$	48.78 $± 0.54$	60.44 $± 0.48$	36.34 $± 0.35$	47.28 $± 0.32$	30.09 $± 0.25$	40.08 $± 0.24$
SimpleShot []	68.87 $± 0.32$	83.69 $± 0.23$	55.22 $± 0.24$	73.58 $± 0.19$	43.05 $± 0.16$	62.64 $± 0.14$	36.78 $± 0.12$	56.22 $± 0.12$
Shape Bias (w/o pc) - SimpleShot	68.74 $± 0.34$	82.57 $± 0.25$	56.12 $± 0.25$	72.80 $± 0.25$	44.83 $± 0.17$	62.41 $± 0.14$	38.94 $± 0.13$	56.38 $± 0.11$
Shape Bias (w/ pc) - SimpleShot	70.96 $± 0.33$	81.33 $± 0.24$	58.47 $± 0.25$	70.81 $± 0.20$	46.96 $± 0.17$	60.3 $± 0.14$	40.59 $± 0.14$	54.00 $± 0.11$
FEAT []	70.66 $± 0.33$	84.13 $± 0.23$	57.15 $± 0.24$	74.29 $± 0.19$	44.84 $± 0.16$	63.65 $± 0.14$	38.43 $± 0.12$	57.42 $± 0.11$
Shape Bias (w/o pc) - FEAT	69.21 $± 0.32$	82.56 $± 0.25$	56.76 $± 0.24$	72.95 $± 0.20$	45.15 $± 0.16$	62.58 $± 0.15$	39.24 $± 0.12$	56.60 $± 0.11$
Shape Bias (w/ pc) - FEAT	71.58 $± 0.34$	81.45 $± 0.25$	59.09 $± 0.25$	71.00 $± 0.20$	47.45 $± 0.17$	59.98 $± 0.15$	41.38 $± 0.12$	53.64 $± 0.11$

Table 5 Table 5 5 Table 5

5Results on image-only and shape-biased low-shot recognition on ShapeNet55-LS. Parenthesis show confidence intervals based on 5K low shot episodes. Bold indicates best performance between a low-shot approach with and without shape bias and underline indicates best overall. Adding shape bias leads to consistent improvement for both FEAT and SimpleShot.

Table 5Results on image-only and shape-biased low-shot recognition on ShapeNet55-LS. Parenthesis show confidence intervals based on 5K low shot episodes. Bold indicates best performance between a low-shot approach with and without shape bias and underline indicates best overall. Adding shape bias leads to consistent improvement for both FEAT and SimpleShot.
Episode Setup $→$	1-shot 5-way	5-shot 5-way	1-shot 10-way	5-shot 10-way	1-shot 20-way	5-shot 20-way
RFS []	65.79 $± 0.32$	80.51 $± 0.23$	52.16 $± 0.20$	69.92 $± 0.10$	40.25 $± 0.10$	58.44 $± 0.08$
ProtoNet []	52.00 $± 0.31$	69.65 $± 0.24$	37.75 $± 0.19$	55.87 $± 0.16$	27.00 $± 0.11$	43.16 $± 0.09$
Triplet	61.07 $± 0.34$	71.43 $± 0.28$	46.89 $± 0.22$	58.37 $± 0.18$	35.09 $± 0.12$	46.20 $± 0.08$
SimpleShot []	66.73 $± 0.32$	80.93 $± 0.22$	53.37 $± 0.21$	70.32 $± 0.16$	41.09 $± 0.12$	59.09 $± 0.08$
Shape Bias (w/o pc) - SimpleShot	67.5 $± 0.34$	81.30 $± 0.23$	54.99 $± 0.23$	71.24 $± 0.17$	43.60 $± 0.13$	61.03 $± 0.08$
Shape Bias (w/ pc) - SimpleShot	69.72 $± 0.32$	80.93 $± 0.24$	57.49 $± 0.21$	70.75 $± 0.16$	46.24 $± 0.12$	60.21 $± 0.08$
FEAT []	67.81 $± 0.32$	80.25 $± 0.23$	54.35 $± 0.22$	70.18 $± 0.16$	42.12 $± 0.12$	59.01 $± 0.08$
Shape Bias (w/o pc)- FEAT	67.78 $± 0.32$	81.45 $± 0.22$	55.69 $± 0.22$	71.74 $± 0.16$	44.44 $± 0.13$	61.46 $± 0.08$
Shape Bias (w/ pc) - FEAT	70.24 $± 0.32$	80.95 $± 0.22$	58.45 $± 0.22$	70.95 $± 0.16$	47.03 $± 0.13$	60.43 $± 0.08$

Prototypical Networks [] is a standard metric-based low shot learning approach, which uses the base class set to create low-shot episodes and learn a feature space that embeds object instances close or far based on visual similarity.

FEAT [] builds on Prototypical Networks by learning an additional set-to-set function implemented as a Transformer [] on top of a cross-entropy pre-trained embedding space to refine the class prototypes used for low-shot classification. FEAT achieves state of the art performance for inductive low-shot learning. Note that FEAT requires separate retraining for each $n$ -way $m$ -shot configuration 5 footnote 5 5 footnote 5 Since none of the datasets have more than 10 classes for validation, the 20 and 30-way evaluations are done using a model trained for 10-way classification..

Triplet We use a simple triplet loss-based approach as a baseline algorithm with access to both image and shape information during training, similar to prior approaches in shape retrieval []. A joint embedding is learned by using triplet loss [], creating positive pairs between image and point cloud features of same objects, and negative pairs between image and shape features from different object instances. Empirically we found that this performs better than using category labels. Inference is done by nearest centroid classification, building class prototypes that contain both appearance and shape information by averaging the individual support features.

4.3 subsection 4.3 4.3 §4.3 <tag close=" ">4.3</tag>Explicit Shape-Bias Improves Image-Based Generalization

We evaluate our method of adding shape bias to low-shot learning algorithms with state of the art low-shot image-only classification algorithms and show that shape bias improves performance in a low data regime. We present results on multiple datasets in Tables , , and where we refer to models as Shape Bias (w/pc) if the shape-biased image encoder uses point cloud information to build prototypes (see Fig (b)) and (wo/pc) if there are no point clouds used to build prototypes for both validation and testing (see Fig (a)). Our approach of introducing shape bias, when trained with $L 1$ and $L 2$ losses improves the performance of image-only low-shot recognition algorithms in the low-data, one-shot learning regime for the SimpleShot and FEAT algorithms by up to 6%-points. For the (w/o pc) models that do not have any explicit shape information in the low-shot phase, we see a smaller one-shot improvement, but good five-shot performance. This indicates that shape bias is useful without any explicit shape information in the low-shot phase, and suggests possible future improvements by using strategies other than averaging to combine image and shape information in the low-shot phase.

Figure 5 Figure 5 5 Figure 5 5Examining the distribution of interclass distances in the mappings learned by minimizing either

L 1

+ L 1 L 2

relative to the reference point cloud embedding shows that adding

L 2

in a better approximation of the shape embedding space on both novel categories and categories seen during training. Figure 5Examining the distribution of interclass distances in the mappings learned by minimizing either

L 1

+ L 1 L 2

relative to the reference point cloud embedding shows that adding

L 2

in a better approximation of the shape embedding space on both novel categories and categories seen during training.

We add shape-bias to SimpleShot by directly using the learned image to shape-mapping $f i$ for nearest class mean classification, whereas for FEAT we train the set-to-set Transformer module on top of $f i$ , fine-tuning the model end-to-end as in the original FEAT design. The object shape embeddings for the low-shot supports are fixed and not trained further. Notice that as the total number of categories (the number of low-shot ways) increases, the improvement in one-shot performance increases. Further, our approach of learning shape bias significantly outperforms the triplet-loss based approach, indicating that first learning an embedding space with point clouds only is a better strategy than joint training with images and point clouds. All experiments for SimpleShot are averaged over 5 runs and for FEAT are averaged over 3 runs, indicating consistent performance improvements. To ensure statistical significance, for all experiments we perform 5K low-shot episodes and report results with 95% confidence intervals. 6 footnote 6 6 footnote 6 For further qualitative and quantitative analysis please refer to the supplement.

4.4 subsection 4.4 4.4 §4.4 <tag close=" ">4.4</tag>Analysis of Pairwise Loss

We perform an analysis to determine the benefit of including the pairwise distance loss $L 2$ . In Figure , we plot the pairwise interclass distances of object instances from categories in the validation set for the learned mapping $f i$ trained either with one loss or both losses (blue and orange respectively), along with the interclass distances in the point cloud embedding that $f i$ is trained to learn. The greater the overall interclass distance, the better, and ideally the pairwise distance distributions are the same between the learned mapping and the point cloud mapping. Just optimizing $L 1$ results in learning a poor mapping on both the training set and the novel classes in the validation set, whereas adding the pairwise term $L 2$ leads to a better approximation of the point cloud embedding. The utility of $L 2$ is also shown in Table , with the significant improvement over just $L 1$ on SimpleShot with shape bias.

4.5 subsection 4.5 4.5 §4.5 <tag close=" ">4.5</tag>Shape Bias and Failure Analysis

To better understand the distinctions between the purely image-based low-shot classifier and the shape-biased low shot classifier, we compute the Pearson correlation ( $< p 0.05$ ) between the accuracy achieved on the same 5K low-shot episodes for the point cloud model and the shape-biased and image-only classifiers (Figure ). The shape-biased low-shot classifier correlates more strongly with the point cloud model across multiple datasets. This is evidence for a qualitative difference beyond classification accuracy between the shape biased and purely image low-shot classifiers. This would not be possible if the image data was such that it could not be classified differently as a result of introducing shape bias. Furthermore, in Table we see that shape-biased SimpleShot misclassifies similarly to the point cloud SimpleShot, and that there is significant room for improvement by learning to map images into shape embeddings more accurately.

Figure 6 Figure 6 6 Figure 6 6Accuracy of point cloud SimpleShot model vs. shape bias and image SimpleShot models over the same 5K episodes shows higher correlation between the point cloud model and shape bias model, indicating that the shape-biased model classifies more similarly to the point cloud model than the image only model. Figure 6Accuracy of point cloud SimpleShot model vs. shape bias and image SimpleShot models over the same 5K episodes shows higher correlation between the point cloud model and shape bias model, indicating that the shape-biased model classifies more similarly to the point cloud model than the image only model. Table 6 Table 6 6 Table 6 6Percent of queries misclassified by shape-biased SimpleShot but not misclassified by point cloud model (over 5K episodes). This indicates there is significant room for improvement by learning better maps from images to shape embeddings.

Table 6Percent of queries misclassified by shape-biased SimpleShot but not misclassified by point cloud model (over 5K episodes). This indicates there is significant room for improvement by learning better maps from images to shape embeddings.
	ModelNet	ShapeNet	Toys
5-way	38.73%	44.81%	57.59%
10-way	30.88%	38.17%	50.53%

5 section 5 5 §5 <tag close=" ">5</tag>Discussion and Conclusion

This paper takes the first step in investigating the utility of shape bias for low-shot object categorization. Through extensive empirical analysis of our novel approach for adding shape bias to image-only low-shot learning algorithms, we demonstrate improved generalization. We also introduce Toys4K, a diverse and challenging dataset for object learning with the largest number of categories available to date. While dependence of our findings on synthetic object data limits our ability to draw conclusions about shape bias under more general conditions, it is essential since it is currently the only feasible way to obtain matched 2D and 3D data at a large enough scale. Moreover, synthetic data has been widely adopted for other vision tasks [].

Progress in few-shot learning is crucial in order to overcome the need for large amounts of labeled training data. This work constitutes a step in a new direction: the exploitation of the natural biases of the visual world, such as object shape, in the design of few-shot architectures. Building on this approach by exploiting other sources of bias is a logical and exciting direction for future work.

6 section 6 6 §6 <tag close=" ">6</tag>Acknowledgement

We would like to thank Rohit Gajawada and Jiayuan Chen for their help with initial data collection. We also thank Zixuan Huang and Miao Liu for their helpful discussion of the paper draft. This work was supported by NSF award 1936970 and NIH award R01-MH114999.

References

Appendix

This supplementary material document is structured as follows: In Section we provide further detail about the training data used in the paper; In section we provide details on the baselines used in the paper, their implementation details and the hyperparameters used for training; In Section we provide empirical evidence about our choice of point-cloud encoding architecture; In Section we provide further details about the training procedure of the shape-biased image embeddings used in the paper.

Appendix A Appendix A A Appendix A <tag close=" ">Appendix A</tag>Further Dataset Details AFurther Dataset Details

In this section we provide details on the composition of the datasets used in the main paper. We provide example images used to illustrate the data used for training in Figure . As a result of using a ray tracing-based renderer Cycles \citeNewblender1, the synthetic image data used for training has high realism. For all algorithms we use $× 224224$ RGB images as input. For point cloud-based learning we use the 3D $(x, y, z)$ coordinates 1024 randomly sampled points as input. For images we use standard geometric data augmentations e.g. flipping, cropping, slight translation and rotation, as well as color jittering, since we found these result in improved validation performance. For point clouds we use the same augmentation procedures as in \citeNewpointnet2-repo, which include translation, jittering and dropout.

A.1 subsection A.1 A.1 §A.1 <tag close=" ">A.1</tag>Toys4K

We provide further details on the composition of our new Toys4K dataset in Table . The 40 train, 10 validation, and 55 test classes split is shown in Table . When performing validation and testing on Toys4K, we generate low-shot episodes consisting of up to 5 shots and 10 queries.

A.2 subsection A.2 A.2 §A.2 <tag close=" ">A.2</tag>ModelNet40-LS

The 20 train, 10 validation, 10 test classes split for ModelNet40-LS is shown in Table . When performing validation and testing on ModelNet40-LS, we generate low-shot episodes consisting of up to 5 shots and 15 queries.

A.3 subsection A.3 A.3 §A.3 <tag close=" ">A.3</tag>ShapeNet55-LS

The 25 train, 10 validation, 20 test classes split for ShapeNet55-LS is shown in Table . When performing validation and testing on ShapeNet55-LS, we generate low-shot episodes consisting of up to 5 shots and 15 queries.

Figure 7 Figure 7 7 Figure 7 7Rendered image samples from multiple categories of ShapeNet, ModelNet and Toys4K. Note the high image quality as a result of using ray-tracing based rendering. Figure 7Rendered image samples from multiple categories of ShapeNet, ModelNet and Toys4K. Note the high image quality as a result of using ray-tracing based rendering. Table 7 Table 7 7 Table 7

7The category composition of the Toys4K dataset.

Table 7The category composition of the Toys4K dataset.
chair	210	tree	57	knife	45	piano	39	shark	30	panda	24	submarine	18
bottle	111	candy	56	trashcan	44	boat	38	stove	29	orange	24	helmet	17
robot	105	guitar	55	ball	44	bread	38	bowl	28	mushroom	23	bicycle	16
dog	103	apple	54	frog	43	fish	37	car	28	phone	23	lion	16
mug	97	flower	54	ice cream	43	horse	36	cookie	28	train	22	motorcycle	16
hammer	94	ladder	53	dragon	43	spade	36	cupcake	28	tv	21	hamburger	16
cat	79	penguin	51	pan	42	banana	35	bunny	27	toaster	21	grapes	16
dinosaur	76	keyboard	51	battery cell	41	airplane	35	drum	26	helicopter	20	tractor	16
deer/moose	65	pencil	50	whale	41	donut	34	pizza	26	lizard	20	monkey	16
fox	64	plate	50	shoe	40	truck	34	mouse	25	saw	19	pc mouse	15
hat	64	key	49	laptop	40	coin	33	chicken	25	marker	19	light bulb	15
sofa	63	chess piece	49	pig	40	snake	32	sink	25	microwave	18	closet	15
glass	63	cake	48	sheep	39	fridge	32	cow	25	bus	18	fries	15
cup	60	screwdriver	46	crab	38	octopus	31	dolphin	25	pear	18	sandwich	15
monitor	57	elephant	46	radio	38	fan	31	violin	25	butterfly	18	giraffe	15

Appendix B Appendix B B Appendix B <tag close=" ">Appendix B</tag>Further Low-Shot Analysis BFurther Low-Shot Analysis

In this section we provide further anaylsis of the low-shot performance by presenting confusion matrices and classification performance in invidividual low-shot episodes.

B.1 subsection B.1 B.1 §B.1 <tag close=" ">B.1</tag>Confusion Matrices

Please refer to Figure and Figure for low-shot confusion matrices on ModelNet40-LS and ShapeNet55-LS. The confusion matrices are obtained by evaluation 5K low-shot episodes for each dataset (10-way for ModelNet40-LS and 20-way for ShapeNet55-LS), and counting how each sample was classified. The confusion matrices reflect the results presented in Section in the main text that adding shape bias improves overall low-shot classification performance.

B.2 subsection B.2 B.2 §B.2 <tag close=" ">B.2</tag>Per-episode Analysis

We provide a per-episode analysis of low-shot classification in Figure to show qualitative evidence of low-shot learning with shape bias. We see that there are cases in which even though there are no view ambiguities, the image-only model misclassifies whereas the shape-biased model correctly classifies (e.g. in the lower left episode, confusing bicycle for sheep).

Appendix C Appendix C C Appendix C <tag close=" ">Appendix C</tag>Baseline Algorithm Details CBaseline Algorithm Details

All algorithms in this paper are implemented using PyTorch \citeNewpaszke2019pytorch. In this section we provide further detail about the baseline implementations and hyperparameters used for training.

C.1 subsection C.1 C.1 §C.1 <tag close=" ">C.1</tag>SimpleShot

The implementation in our codebase for SimpleShot \citeNewwang2019simpleshot1 is based on the code release by the authors in \citeNewsimpleshot-repo. The authors report a 1-shot 5-way accuracy of $⁢ 49.69 (0.19)$ and a 5-shot 5-way accuracy of $⁢ 66.92 (0.17)$ on miniImageNet \citeNewvinyals2016matching1 with the Conv4 architecture. The reimplementation of SimpleShot in our codebase with the same dataset and architecture results in 1-shot 5-way accuracy of $⁢ 50.60 (0.34)$ and a 5-shot 5-way accuracy of $⁢ 68.06 (0.23)$ .

In all our experiments we train SimpleShot with SGD with an initial learning rate of $0.01$ and a learning rate decay of $0.1$ at epochs 300 and 360, out of a total of 400 epochs. SimpleShot employs three different feature normalization strategies, no normalization, $L 2$ normalization and $L 2$ normalization and training set mean subtraction. In experiments with SimpleShot we report the result of the best of these three normalization strategies.

C.2 subsection C.2 C.2 §C.2 <tag close=" ">C.2</tag>RFS

The implementation in our codebase for RFS \citeNewtian2020rethink1 is based on the code release by the authors in \citeNewRFS-repo. The original codebase obtains a 1-shot 5-way accuracy of $⁢ 53.73 (0.81)$ on miniImageNet \citeNewvinyals2016matching1 with the Conv4 architecture. The reimplementation of RFS in our codebase with the same dataset and architecture results in 1-shot 5-way accuracy of $⁢ 54.59 (0.86)$ . RFS requires training an embedding on the training dataset using cross-entropy. We train this embedding space with SGD using a learning rate of 0.001, momentum of 0.9 and $L 2$ weight penalty weight parameter of 0.0005. For each low-shot episode we train a logistic regression classifier using Scikit-learn\citeNewscikit-learn, as in the original RFS.

C.3 subsection C.3 C.3 §C.3 <tag close=" ">C.3</tag>FEAT

The implementation for FEAT is based on the code release by the authors in \citeNewFEAT-repo. The original codebase obtains a 1-shot 5-way accuracy of $⁢ 54.85 (0.20)$ and 5-shot 5-way accuracy of $71.61$ on miniImageNet \citeNewvinyals2016matching1 with the Conv4 architecture. The reimplementation of FEAT in our codebase with the same dataset and architecture results in 1-shot 5-way accuracy of $⁢ 54.85 (0.20)$ 5-shot 5-way accuracy of $⁢ 71.45 (0.73)$ . We train FEAT with the default hyperparameters recommended by the authors, training separate models for 5-way and 10-way classification, and separate models for 1-shot and 5-shot, as recommended by the authors. For the shape biased FEAT we do not use learning rate scheduling and momentum, since they have a negative effect on performance for shape-biased training. Removing them for image-only training does not affect performance.

Figure 8 Figure 8 8 Figure 8 8Confusion matrices over 5K low-shot episodes of SimpleShot for Image Only, Shape-Biased without access to point clouds (w/o pc) at test time and Shape-Biased with (w/ pc) access to point clouds at test time on the ModelNet-LS dataset. Even without access to point clouds (w/o pc) for building class prototypes, the shape-biased image embedding leads to improvements. Adding point cloud support information (w/ pc) improves performance further. See Table for aggregate results. Figure 8Confusion matrices over 5K low-shot episodes of SimpleShot for Image Only, Shape-Biased without access to point clouds (w/o pc) at test time and Shape-Biased with (w/ pc) access to point clouds at test time on the ModelNet-LS dataset. Even without access to point clouds (w/o pc) for building class prototypes, the shape-biased image embedding leads to improvements. Adding point cloud support information (w/ pc) improves performance further. See Table for aggregate results.

Figure 9 Figure 9 9 Figure 9 9Confusion matrices over 5K low-shot episodes of SimpleShot for Image Only, Shape-Biased without access to point clouds (w/o pc) at test time and Shape-Biased with (w/ pc) access to point clouds at test time on the ShapeNet-LS dataset. As in ModelNet40-LS, without access to point clouds (w/o pc) for building class prototypes, the shape-biased image embedding leads to improvements. Adding point cloud support information (w/ pc) improves performance further. See Table for aggregate results. Best viewed with zoom. Figure 9Confusion matrices over 5K low-shot episodes of SimpleShot for Image Only, Shape-Biased without access to point clouds (w/o pc) at test time and Shape-Biased with (w/ pc) access to point clouds at test time on the ShapeNet-LS dataset. As in ModelNet40-LS, without access to point clouds (w/o pc) for building class prototypes, the shape-biased image embedding leads to improvements. Adding point cloud support information (w/ pc) improves performance further. See Table for aggregate results. Best viewed with zoom. C.4 subsection C.4 C.4 §C.4 <tag close=" ">C.4</tag>Protoypical Networks

The implementation in our codebase for Prototypical Networks is based on the code release by the SimpleShot authors in \citeNewsimpleshot-repo. In \citeNewchen2018closer the authors report that their reimplementation obtains a 5-shot 5-way accuracy of $⁢ 66.68 (0.68)$ on miniImageNet \citeNewvinyals2016matching1 with the Conv4 architecture. The reimplementation of Prototypical Networks in our codebase with the same dataset and architecture results in 5-shot 5-way accuracy of $⁢ 66.94 (0.71)$ . We train separate Prototypical Networks models for 5-shot classification and 1-shot classification. As recommended by the authors of the original paper, we perform 20-way training. We use the Adam \citeNewkingma2014adam optimizer, 400 low-shot iterations per epoch, 200 epochs total, and a learning rate of 0.0001 $L 2$ and weight penalty weight parameter of 0.00001. We perform a learning rate decay of 0.5 every 20 epochs.

Figure 10 Figure 10 10 Figure 10 10Six low-shot episodes for 5 ways, 1 shot and 1 query on Toys4K for shape-biased SimpleShot. We visually display the composition of the image and point cloud shots and the image queries, as well as the models’ predictions, illustrating cases where shape bias allows for improved performance. Best viewed with zoom. Figure 10Six low-shot episodes for 5 ways, 1 shot and 1 query on Toys4K for shape-biased SimpleShot. We visually display the composition of the image and point cloud shots and the image queries, as well as the models’ predictions, illustrating cases where shape bias allows for improved performance. Best viewed with zoom. C.5 subsection C.5 C.5 §C.5 <tag close=" ">C.5</tag>Triplet Model

We implement a joint triplet model using both point cloud (DGCNN \citeNewwang2019dynamic1 and image (ResNet18 \citeNewhe2016deep1) encoders, which can use both image and shape information during training. Let $f i$ denote the image encoder, $f p$ denote the point cloud encoder, and $ϕ p k$ and $ϕ i k$ denote the point cloud and image encodings of object instance $k$ respectively. We learn a joint image/shape embedding by minimizing a standard triplet loss

= ⁢ L (ϕ i k, ϕ p k, ϕ p l) max {+ - ⁢ d (ϕ i k, ϕ p k) ⁢ d (ϕ p k, ϕ p l) margin, 0}

⁢ L (ϕ i k, ϕ p k, ϕ p l)

= max {+ - ⁢ d (ϕ i k, ϕ p k) ⁢ d (ϕ p k, ϕ p l) margin, 0}

= ⁢ d (x, y) || - x y || 2

⁢ d (x, y)

= || - x y || 2

where the anchor is an image embedding of instance $k$ , $ϕ i k$ , the positive sample is a point cloud encoding of the same object instance, $ϕ i k$ , and the negative sample is a $ϕ p l$ is a point cloud embedding of a different object instance $l$ . We perform $L 2$ normalization of the embeddings prior to computing the loss. Note that it is possible to build (anchor, positive, negative) pairs using category information, but we empirically found that this leads to worse performance.

We train the triplet model using the Adam \citeNewkingma2014adam optimizer with a learning rate of 0.0001, $L 2$ weight penalty weight parameter of 0.0001, and margin of 0.1. We use a batch size of 72 and train for 600 epochs, each epoch consisting of 20K random samples.

Appendix D Appendix D D Appendix D <tag close=" ">Appendix D</tag>Learning a Point Cloud Shape Embedding DLearning a Point Cloud Shape Embedding

In this section we describe the algorithm for learning a point-cloud based embedding space, and present an empirical study for our point cloud architecture choice.

D.1 subsection D.1 D.1 §D.1 <tag close=" ">D.1</tag>Algorithm

The algorithm we use to train a point-cloud embedding space is based on SimpleShot \citeNewwang2019simpleshot1 and is described with pseudocode in Algorithm . Note that the routine AccAccumulator is used denotes a function to collect the validation accuracy of each low-shot episode and compute summary statistics. The NNClassify routine takes support features and labels, and classifies each test query feature based on a nearest neighbors rule using cosine similarity. The point-cloud embedding model is trained using SGD with a learning rate of 0.01, batch size of 129, and $L 2$ weight penalty weight parameter of 0.0001. We perform learning rate decay by 0.1 at epochs 300 and 360. In all models we use features from the output of the pooling layer in the architecture.

D.2 subsection D.2 D.2 §D.2 <tag close=" ">D.2</tag>Architecture Study

We perform an empirical study on the point cloud architectures to determine which is capable of the best low-shot generalization performance. Our PointNet \citeNewqi2017pointnet1 implementation is based on \citeNewpointnet-repo, our PointNet++ \citeNewqi2017pointnet++1 is based on \citeNewpointnet2-repo and our DGCNN \citeNewwang2019dynamic1 implementation is based on \citeNewdgcnn-repo. We use a DGCNN architecture with a reduced embedding dimension (size after the pooling operation) of 512 rather than the original 1024, to match the dimensionality of the ResNet18 embeddings. We find no decrease in performance by this reduction. We present the results of this study on ModelNet in Table . The DGCNN \citeNewwang2019dynamic1 architecture outperforms other point cloud architectures at low-shot generalization to novel categories. We find that randomly rotating the input point cloud about the origin during training (random rotation about all axes of rotation, indicated by SO3 in the table) results in a performance improvement. We use this SO3 strategy for all shape-embedding space learning experiments.

Table 8 Table 8 8 Table 8

8Empirical study for choosing the best point cloud architecture. Reported is 1-shot 5-way classification accuracy on the ModelNet40-LS validation set. We find that DGCNN performs the best, and that randomly rotating each input point cloud during training (indicated with SO3) results in a improvement in low-shot generalization performance as well.

Table 8Empirical study for choosing the best point cloud architecture. Reported is 1-shot 5-way classification accuracy on the ModelNet40-LS validation set. We find that DGCNN performs the best, and that randomly rotating each input point cloud during training (indicated with SO3) results in a improvement in low-shot generalization performance as well.
Architecture	1-shot 5-way accuracy
PointNet \citeNewqi2017pointnet1	66.13
PointNet++ \citeNewqi2017pointnet++1	67.49
DGCNN \citeNewwang2019dynamic1	75.2
DGCNN (SO3)	77.5

Appendix E Appendix E E Appendix E <tag close=" ">Appendix E</tag>Details for Learning a Shape Biased Image Embedding EDetails for Learning a Shape Biased Image Embedding

The algorithm we use to train a shape-biased image embedding is described with pseudocode in Algorithm . We use the Adam optimizer with a batch size of 256, an initial learning rate of 0.001 and a $L 2$ weight penalty weight parameter of 0.0001. The model is trained for 400 epochs, with a learning rate decay of 0.1 at epochs 300 and 360.

E.1 subsection E.1 E.1 §E.1 <tag close=" ">E.1</tag>SimpleShot with Shape Bias

The SimpleShot \citeNewwang2019simpleshot1 approach does not require any learning (parameter updates) during the low-shot phase. Classification is done using nearest centroid classification in the embedding space. The image embedding function $f i$ is trained as described in Algorithm , and low-shot testing is done following the same procedure as described in L8-16 in Algorithm but using nearest centroid rather than nearest neighbor classification.

E.2 subsection E.2 E.2 §E.2 <tag close=" ">E.2</tag>FEAT with Shape Bias

The algorithm we use to train a shape-biased FEAT \citeNewYe_2020_CVPR1 architecture is described in Algorithm . Note that the $f i$ used in this algorithm is being fine tuned from a mapping already trained with Algorithm while the FEAT set-to-set function $E$ is trained from scratch. For this experiment we use the default hyperparameters recommended by the FEAT authors. Low shot testing is done following the same procedure as described in L13-22 in Algorithm but using the test set. The procedure we refer to as FEATClassify is described in Eq. 4 on pg. 4 of the FEAT paper \citeNewYe_2020_CVPR1. In the pseudocode FEATClassify performs classification and directly outputs the per-episode classification accuracy.

\algorithmcfname 1 1 1 \SetAlgoLined\KwInRandomly initialized point-cloud classifier architecture

f p

with embedding function

f p E

Total number of epochs

N e

Total number of mini-batches per epoch

N b

Total number of low-shot iterations for validation

N ⁢ i t

\KwData(point cloud, label) pair datasets

D train

D val

\KwDef

: ℓ

cross-entropy loss \ForEachepoch in

1, 2, …, N e

\ForEachmini-batch

∼ (o p, y) D train

N b

Predict

=^y ⁢ f p (o p)

Compute

⁢ ℓ (y,^y)

Compute

∇ ℓ

with respect to

f p

Update

f p

with SGD A = AccAccumulator \ForEachvalidation episode in

1, 2, …, N e

Sample

5

-way

1

-shot

∼ (o p train, y train, o p test) D val

Predict

= ϕ p train ⁢ f p E (o p train)

Predict

= ϕ p test ⁢ f p E (o p test)

acc = NNClassify(

ϕ p train, y train, ϕ p test

) A(acc) val accuracy = A.average() \Ifval accuracy

>

best accuracy best acuracy

←

val accuracy

← f p best f p

\KwResultTrained

f p best

1Training Shape Embedding

f p

\algorithmcfname 1Training Shape Embedding

f p

\algorithmcfname 2 2 2 \SetAlgoLined\KwInRandomly initialized image embedding architecture

f i

Point-cloud embedding function

f p

()Total number of epochs

N e

Total number of mini-batches per epoch

N b

Total number of low-shot iterations for validation

N ⁢ i t

\KwData(image, point cloud, label) pair datasets

D train

D val

\KwDef

= L + L 1 L 2

(see main text for def.) \ForEachepoch in

1, 2, …, N e

\ForEachmini-batch

∼ (o i, o p, y) D train

N b

Predict shape embedding

= ϕ p ⁢ f p (o p)

Predict image embedding

= ϕ i ⁢ f i (o i)

Compute

L

using

ϕ p

and

ϕ i

Compute

∇ L

with respect to

f i

Update

f i

with Adam A = AccAccumulator \ForEachvalidation episode in

1, 2, …, N e

Sample

5

-way

1

-shot

∼ (o i train, o p train, y train, o i test) D val

Predict

= ϕ p train ⁢ f p (o p train)

Predict

= ϕ i train ⁢ f i (o i train)

Predict

= ϕ i test ⁢ f i (o i test)

← ϕ train

Average

(ϕ p train, ϕ i train)

acc = NNClassify(

ϕ train, y train, ϕ i test

) A(acc) val accuracy = A.average() \Ifval accuracy

>

best accuracy best acuracy

←

val accuracy

← f i best f i

\KwResultTrained

f i best

2Training Shape-Biased Image Embedding Function

f i

\algorithmcfname 2Training Shape-Biased Image Embedding Function

f i

\algorithmcfname 3 3 3 \SetAlgoLined\KwInShape-biased image encoder

f i

()Point-cloud embedding function

f p

()Randomly initialized FEAT \citeNewYe_2020_CVPR1 set-to-set function

E

–see p3 in \citeNewYe_2020_CVPR1.Total number of epochs

N e

Total number of low-shot iterations per training epoch

N ⁢ i t

Total number of low-shot iterations for validation

N - v ⁢ i t

\KwData(image, point cloud, label) pair datasets

D train

D val

\KwDef

L FEAT

– Eq. 7 in \citeNewYe_2020_CVPR1 \ForEachepoch in

1, 2, …, N e

\ForEachtraining episode in of

1, 2, …, N ⁢ i t

Sample

m

-way

n

-shot

∼ (o i train, o p train, o p query, y train, y query) D train

Predict ptcld. support

= ϕ p train ⁢ f p (o p train)

Predict image support

= ϕ i train ⁢ f i (o i train)

Predict image queries

= ϕ i query ⁢ f i (o i query)

← ϕ train

Average

(ϕ p train, ϕ i train)

←^ϕ train,^ϕ i query ⁢ E (ϕ train, ϕ i query)

Compute

L

using

^ϕ train,^ϕ i query

and

y train, y query

Compute

∇ L

with respect to

f i

and

E

Update

f i, E

with SGD A = AccAccumulator \ForEachvalidation episode in

1, 2, …, N - v ⁢ i t

Sample

5

-way

1

-shot

∼ (o i train, o p train, y train, o i test) D val

Predict ptcld. support

= ϕ p train ⁢ f p (o p train)

Predict image support

= ϕ i train ⁢ f i (o i train)

Predict image queries

= ϕ i test ⁢ f i (o i test)

← ϕ train

Average

(ϕ p train, ϕ i train)

acc = FEATClassify(

ϕ train, y train, ϕ i test

) A(acc) val accuracy = A.average() \Ifval accuracy

>

best accuracy best acuracy

←

val accuracy

← f i best f i

← E best E

\KwResultTrained

f i best, E best

3Training FEAT with Shape Bias \algorithmcfname 3Training FEAT with Shape Bias Table 9 Table 9 9 Table 9

9Split composition of ShapeNet55-LS

Table 9Split composition of ShapeNet55-LS
Training	# samples	Validation	# samples	Testing	# samples
vessel	873	train	389	mug	214
car	530	bed	233	tower	133
sofa	500	stove	218	motorcycle	337
lamp	500	bowl	186	cap	56
cellular	500	pillow	96	pistol	307
faucet	500	mailbox	94	earphone	73
pot	500	rocket	85	skateboard	152
guitar	500	birdhouse	73	camera	113
airplane	500	microphone	67	piano	239
bus	500	keyboard	65	printer	166
chair	500			bag	83
rifle	500			trashcan	343
cabinet	500			file	298
bench	499			dishwasher	93
bathtub	499			microwave	152
telephone	499			washer	169
jar	499			remote	66
bottle	498			helmet	162
display	496			basket	113
clock	496			can	108
loudspeaker	496
table	495
laptop	460
bookshelf	452
knife	423
Total
25 classes	12716	10 classes	1506	20 classes	3377

Table 10 Table 10 10 Table 10

10Split composition of ModelNet40-LS

Table 10Split composition of ModelNet40-LS
Training	# samples	Validation	# samples	Testing	# samples
bed	615	cup	99	range hood	215
car	297	xbox	123	bowl	84
guitar	255	bathtub	156	stool	110
bottle	435	cone	187	radio	124
desk	286	curtain	158	stairs	144
night stand	286	door	129	lamp	144
glass box	271	flower pot	169	tent	183
sofa	780	person	108	sink	148
piano	331	wardrobe	107	bench	193
toilet	444	keyboard	165	laptop	169
monitor	565
table	492
dresser	286
airplane	726
tv stand	367
chair	989
bookshelf	672
vase	575
plant	340
mantel	384
Total
20 classes	9396	10 classes	1401	10 classes	1514

Table 11 Table 11 11 Table 11

11Split composition of Toys4K

Table 11Split composition of Toys4K
Training	# samples	Validation	# samples	Testing	# samples
candy	56	airplane	35	boat	38
flower	54	shark	30	lion	17
dragon	43	truck	34	whale	41
apple	54	phone	23	cupcake	28
guitar	55	giraffe	15	train	22
tree	57	horse	37	pizza	26
glass	63	fish	37	marker	19
cup	60	fan	31	cookie	28
pig	41	shoe	41	sandwich	15
cat	79	snake	32	octopus	31
chair	210			monkey	16
ice cream	43			fries	15
hat	64			violin	25
deer moose	65			mushroom	23
penguin	53			closet	15
ball	44			tractor	16
fox	64			submarine	18
dog	103			butterfly	18
knife	45			pear	18
laptop	41			bicycle	17
pen	42			dolphin	25
mug	97			bunny	27
plate	50			coin	33
chess piece	49			radio	40
cake	48			grapes	16
frog	43			banana	35
ladder	53			cow	25
keyboard	51			donut	34
sofa	63			stove	29
trashcan	44			sink	25
dinosaur	76			orange	24
bottle	111			saw	19
elephant	46			chicken	25
pencil	50			hamburger	16
key	49			piano	39
monitor	57			light bulb	15
hammer	94			spade	36
screwdriver	46			crab	40
robot	105			sheep	40
bread	38			toaster	21
				lizard	20
				motorcycle	16
				mouse	25
				pc mouse	15
				bus	18
				helicopter	20
				microwave	18
				cells battery	41
				drum	26
				panda	24
				tv	21
				car	28
				helmet	17
				fridge	31
				bowl	28
Total
40 classes	2506	10 classes	315	55 classes	1358

\bibliographystyleNew

ieee_fullname \bibliographyNewegbib_supp