BOP Challenge 2019/2020

  • 11/Aug/2020 - The BOP Challenge 2020 deadline has been extended to August 19 (11:59PM UTC).
  • 05/Jun/2020 - BOP Challenge 2020 has been opened!
  • 27/Jan/2020 - Submissions to the BOP Challenge 2019 have been re-evaluated.
  • 28/Oct/2019 - The winners of the BOP Challenge 2019 have been announced! The submission form stays open and snapshots of the leaderboard will be presented and discussed at the next R6D workshops. A report with an analysis of the BOP Challenge 2019 results is in preparation.
  • 26/Jul/2019 - BOP Challenge 2019 has been opened.

Join the BOP Google group to stay up to date.

1. Introduction

This page defines the 2019 and 2020 editions of the BOP Challenge. The two editions share the same evaluation methodology and the same leaderboard. The winners of the 2019 edition were announced on October 28, 2019 at the 5th Workshop on Recovering 6D Object Pose (ICCV 2019). The 2020 edition includes a new photorealistic renderer and a set of pre-rendered training images, which reduces the entry barrier of the challenge for learning-based RGB and RGB-D methods. The winners of the 2020 edition will be announced on August 23, 2020 at the 6th Workshop on Recovering 6D Object Pose (ECCV 2020).

Please cite the ECCV 2018 paper [1] (PDF, BIB) when referring to the BOP Challenge and [8,1] (PDF, BIB) when referring to the BlenderProc4BOP renderer.

What's new in the BOP Challenge 2020

Photorealistic training images: In 2020, the challenge is focused on the synthesis of effective RGB training images. While learning from synthetic data has been common for depth-based pose estimation methods, the same is still difficult for RGB-based methods where the domain gap between synthetic training and real test images is more severe. Specifically for the challenge, we have therefore prepared BlenderProc4BOP, an open-source and light-weight photorealistic (PBR) renderer, and used it to render training images for each of the seven core datasets (example images are below). With this addition, we hope to reduce the entry barrier of the challenge for participants working on learning-based RGB and RGB-D solutions. We are excited to see whether photorealistic training images will help to close the performance gap between depth-based and RGB-based methods that we have observed in the previous iteration of the challenge. Participants are encouraged to build on top of the renderer and release their extensions.

BOP photorealistic training imageBOP photorealistic training imageBOP photorealistic training imageBOP photorealistic training imageBOP photorealistic training image

Short papers in the ECCV 2020 workshop proceedings: Participants of the BOP Challenge 2020 have the opportunity to document their methods by submitting a short paper to the 6th Workshop on Recovering 6D Object Pose (ECCV 2020). The paper will be reviewed by the organizational committee and accepted if it presents a method with competitive results or distinguishing features. The paper must have exactly 4 pages including references and, if accepted, will be published in the ECCV workshop proceedings. Note that besides the short papers, the workshop invites submissions of full papers (14 pages excluding references) on any topic related to 6D object pose estimation.

Evaluation methodology as in 2019: The BOP Challenge 2020 follows the same evaluation methodology as the BOP Challenge 2019. The scores have not been saturated and following the same methodology allows using the results from 2019 as baselines.

Submission deadlines:
Results to the BOP Challenge 2020: August 16 August 19, 2020 (11:59PM UTC).
Short papers documenting the methods: August 16 August 19, 2020 (11:59PM UTC). Submit the papers by e-mail to r6d.workshop@gmail.com.

2. Task definition

The challenge is on the task of 6D localization of a varying number of instances of a varying number of objects in a single RGB-D image (the ViVo task).

Training input: At training time, a method learns using a training set, $T = \{T_o\}$, where $o$ is an object identifier. Training data $T_o$ may have different forms – a 3D mesh model of the object or a set of RGB/RGB-D/D images (synthetic or real) showing object instances in known 6D poses.

Test input: At test time, the method is provided with image $I$ and list $L = [(o_1, n_1), ..., (o_m, n_m)]$, where $n_i$ is the number of instances of object $o_i$ present in image $I$.

Test output: The method produces list $E = [E_1, \dots, E_m]$, where $E_i$ is a list of $n_i$ pose estimates for instances of object $o_i$. Each estimate is given by a 3x3 rotation matrix, $\mathbf{R}$, a 3x1 translation vector, $\mathbf{t}$, and a confidence score, $s$.

The ViVo task is referred to as the 6D localization problem in [2]. In the BOP paper [1], methods were evaluated on a different task – 6D localization of a single instance of a single object (the SiSo task), which was chosen because it allowed to evaluate all relevant methods out of the box. Since then, the state of the art has advanced and we have moved to the more challenging ViVo task.

3. Datasets

Multiple datasets are used for the evaluation. Each dataset is provided in the BOP format and includes 3D object models and training and test RGB-D images annotated with ground-truth 6D object poses. Some datasets (HB, ITODD) include also validation images – in this case, the ground-truth poses are publicly available only for the validation images, not for the test images. The object models were created manually or using KinectFusion-like systems for 3D surface reconstruction [6]. The training images were either captured by an RGB-D/Gray-D sensor or obtained by rendering of the object models. The test images were captured in scenes with graded complexity, often with clutter and occlusion.

For training, a method can use the provided object models and training images. It can also render extra training images using the models. Not a single pixel of test images may be used for training, nor the individual ground-truth poses or object masks provided for the test images. The range of all ground-truth poses in the test images, which is provided in file dataset_params.py in the BOP toolkit (items depth_range, azimuth_range, and elev_range), is the only information about the test set that can be used for training.

3.1 List of datasets

Core datasets: LM-O, T-LESS, TUD-L, IC-BIN, ITODD, HB, YCB-V. A method needs to be evaluated on these seven datasets to be considered for the main awards.
Other datasets: LM, RU-APC, IC-MI, TYO-L.

Only subsets of the original datasets are used to speed up the evaluation. The subsets are defined in files test_targets_bop19.json provided with the datasets.

3.2 Photorealistic training images

For the BOP Challenge 2020, we provide 50K photorealistic training images for each of the seven core datasets: LM/LM-O, T-LESS, TUD-L, IC-BIN, ITODD, HB, YCB-V. The images were rendered with BlenderProc4BOP, an open-source and light-weight photorealistic (PBR) renderer prepared for the BOP Challenge. The renderer implements a synthesis approach similar to [7]. The main difference is that the objects are not rendered in 3D models of complete indoor scenes but inside a cube, which makes the rendering noticeably faster. A rich spectrum of the background is ensured by assigning a random PBR material from the CC0 Textures library to the walls of the cube.

4. Awards

  1. The Overall Best Method – The top-performing method on the seven core datasets.
  2. The Best RGB-Only Method – The top-performing RGB-only method on the seven core datasets.
  3. The Best Fast Method – The top-performing method on the seven core datasets with the average running time per image below 1s.
  4. The Best BlenderProc4BOP-Trained Method – The top-performing method on the seven core datasets which was trained only with the provided BlenderProc4BOP images.
  5. The Best Single-Model Method – The top-performing method on the seven core datasets which uses a single machine learning model (typically a neural network) per dataset.
  6. The Best Open-Source Method – The top-performing method on the seven core datasets whose source code is publicly available.
  7. The Best Method on Dataset D – The top-performing method on each of the eleven available datasets.

See conditions which a method needs to fulfill in order to qualify for the awards.

The BOP Challenge 2019 awards were presented at the 5th Workshop on Recovering 6D Object Pose (ICCV 2019, Seoul).
The BOP Challenge 2020 awards will be presented at the 6th Workshop on Recovering 6D Object Pose (ECCV 2020, Glasgow).

5. How to participate

To have your method evaluated, run it on the ViVo task and submit the results in the format described below to the evaluation system (the evaluation script used in the system is publicly available in the BOP Toolkit). Note that each method has to use an identical set of hyper-parameters across all objects and datasets.

The list of object instances for which the pose is to be estimated can be found in files test_targets_bop19.json provided with the datasets. For each object instance in the list, at least $10\%$ of the object surface is visible in the image [1].

5.1 Format of results

Results for all test images from one dataset are saved in one CSV file, with one pose estimate per line in the following format:

scene_id,im_id,obj_id,score,R,t,time

$\mathbf{P} = \mathbf{K} [\mathbf{R} \, \mathbf{t}]$ is the camera matrix which transforms a 3D point in the model coordinate system, $\mathbf{x}_m = [x, y, z, 1]'$, to a 2D point in the image coordinate system, $\mathbf{x}_i = [u, v, 1]'$: $s\mathbf{x_i} = \mathbf{P} \mathbf{x}_m$. The camera coordinate system is defined as in OpenCV with the camera looking along the $Z$ axis. The intrinsic matrix $\mathbf{K}$ is provided with the test images – note it may be different for each image.

Example results can be found here.

5.2 Terms & conditions

  1. To be considered for the awards and for inclusion in a publication about the challenge, the authors need to provide documentation of the method (including specifications of the used computer) through the online submission form. The authors are also encouraged to document their method in a short paper submitted to the 6th Workshop on Recovering 6D Object Pose at ECCV 2020. The paper must have exactly 4 pages including references and, if accepted, will be published in the ECCV workshop proceedings.
  2. The winners need to present their methods at the awards reception (via an oral presentation in the case of a physical reception and a pre-recorded 5 min video in the case of a virtual reception).
  3. After the submitted results are evaluated (by the online evaluation system), the authors can decide whether to make the scores visible to the public.
  4. A publication about the challenge may include further analysis of the results (only those marked as visible to the public).

6. Evaluation methodology

6.1 Pose-error functions

The error of an estimated pose $\hat{\textbf{P}}$ w.r.t. the ground-truth pose $\bar{\textbf{P}}$ of an object model $O$ is measured by three pose-error functions defined below. Their implementation is available in the BOP Toolkit.

Visible Surface Discrepancy (VSD) [1,2]:

$ e_\mathrm{VSD}(\hat{S}, \bar{S}, S_I, \hat{V}, \bar{V}, \tau) = \mathrm{avg}_{p \in \hat{V} \cup \bar{V}} \begin{cases} 0 & \text{if} \, p \in \hat{V} \cap \bar{V} \, \wedge \, |\hat{S}(p) - \bar{S}(p)| < \tau \\ 1 & \text{otherwise}, \end{cases} $

where $\hat{S}$ and $\bar{S}$ are distance maps obtained by rendering the object model $O$ in the estimated pose $\hat{\textbf{P}}$ and the ground-truth pose $\bar{\textbf{P}}$ respectively. The distance maps are compared with the distance map $S_I$ of the test image $I$ to obtain the visibility masks $\hat{V}$ and $\bar{V}$, i.e. the sets of pixels where the model $O$ is visible in image $I$. Compared to [1,2], the estimation of visibility masks has been modified – at pixels with no depth measurements, an object is now considered visible (it was considered not visible in [1,2]). This modification allows evaluating poses of glossy objects from the ITODD dataset whose surface is not always captured by the depth sensors. $\tau$ is a misalignment tolerance.

VSD treats indistinguishable poses as equivalent by considering only the visible object part. See Section 2.2 of [1] and FAQ for details.

Maximum Symmetry-Aware Surface Distance (MSSD) [3]:

$ e_{\text{MSSD}} = \text{min}_{\textbf{S} \in S_O} \text{max}_{\textbf{x} \in V_O} \Vert \hat{\textbf{P}}\textbf{x} - \bar{\textbf{P}}\textbf{S}\textbf{x} \Vert_2, $

where $S_O$ is a set of global symmetry transformations and $V_O$ is a set of mesh vertices of object model $O$ (see Section 5.2).

The maximum distance is relevant for robotic manipulation, where the maximum surface deviation strongly indicates the chance of a successful grasp. Moreover, compared to the average distance used in ADD/ADI [2, 5] which tends to be dominated by higher-frequency surface parts, the maximum distance is less dependent on the sampling of mesh vertices.

Maximum Symmetry-Aware Projection Distance (MSPD):

$ e_{\text{MSPD}} = \text{min}_{\textbf{S} \in S_O} \text{max}_{\textbf{x} \in V_O} \Vert \text{proj}( \hat{\textbf{P}}\textbf{x} ) - \text{proj}( \bar{\textbf{P}}\textbf{S}\textbf{x} ) \Vert_2, $

where $\text{proj}$ is the 2D projection operation (the result is in pixels) and the meaning of the other symbols is as in MSSD.

Compared to the 2D projection [4], MSPD considers global object symmetries and replaces the average by the maximum distance to increase robustness against the sampling of mesh vertices. Since MSPD does not evaluate the alignment along the optical axis (Z axis) and measures only the perceivable discrepancy, it is relevant for augmented reality applications and suitable for evaluating RGB-only methods.

6.2 Identifying global object symmetries

The set of global symmetry transformations of an object (used in MSSD and MSPD) is identified in two steps. Firstly, we obtain a set of candidate symmetry transformations as $S'_O = \{\textbf{S}: h(V_O, \textbf{S}V_O) < \varepsilon \}$, where $h$ is the Hausdorff distance calculated between vertices $V_O$ of object model $O$ in the canonical and transformed locations. The allowed deviation is bounded by $\varepsilon = \text{max}(15\,mm, 0.1d)$, where $d$ is the diameter of model $O$ (the largest distance between any pair of model vertices) and the truncation at $15\,mm$ avoids breaking the symmetries by too small details. Secondly, we identify the final set of symmetry transformations $S_O$ as a subset of $S'_O$ which consists of those symmetry transformations that cannot be resolved by the model texture (decided subjectively by the organizers of the challenge).

Set $S_O$ covers both discrete and continuous global rotational symmetries. The continuous rotational symmetries are discretized such as the vertex which is the furthest from the axis of symmetry travels not more than $1\%$ of the object diameter between two consecutive rotations. The symmetry transformations are stored in files models_info.json provided with the datasets.

6.3 Performance score

An estimated pose is considered correct w.r.t. pose-error function $e$ if $e < \theta_e$, where $e \in \{e_{\text{VSD}}, e_{\text{MSSD}}, e_{\text{MSPD}}\}$ and $\theta_e$ is the threshold of correctness. The fraction of annotated object instances, for which a correct pose is estimated, is referred to as recall. The Average Recall w.r.t. function $e$, denoted as $\text{AR}_e$, is defined as the average of the recall rates calculated for multiple settings of threshold $\theta_e$, and also for multiple settings of the misalignment tolerance $\tau$ in the case of $e_{\text{VSD}}$.

The performance of a method on a dataset is measured by the Average Recall: $\text{AR} = (\text{AR}_{\text{VSD}} + \text{AR}_{\text{MSSD}} + \text{AR}_{\text{MSPD}}) / 3$. The overall performance on the core datasets is measured by $\text{AR}_{\text{Core}}$ defined as the average of the per-dataset $\text{AR}$ scores. In this way, each dataset is treated as a separate sub-challenge which avoids $\text{AR}_{\text{Core}}$ being dominated by larger datasets.

7. FAQ

7.1 Why synthetic training images?

While 3D object models are often available or can be generated at a low cost using KinectFusion-like systems for 3D surface reconstruction [6], capturing and annotating real training images requires a significant effort. The BOP Challenge is therefore focused primarily on the more practical scenario where only the object models, which can be used to render synthetic training images, are available at training time.

7.2 6D localization vs. 6D detection

Prior information about the object presence in the input image distinguishes two 6D object pose estimation tasks: 6D localization, where the identifiers of present object instances are provided for each image, and 6D detection, where no prior information is provided [2].

The aspect which is evaluated on the 6D detection but not on the 6D localization task is the capability of the method to calibrate the predicted confidence scores across all object classes. For example, a score of $0.5$ for a cat should represent the same level of confidence as a score of $0.5$ for a duck. This calibration is important for achieving good performance w.r.t. the precision/recall curve which is typically used for evaluating detection. The 6D localization task still requires the method to sort the hypotheses, although only within the same object class – the method needs to output the top N pose estimates for an object class which are evaluated against N ground-truth poses of that class.

There are two reasons why the methods are evaluated on the 6D localization task in BOP:

  1. 6D detection has computationally expensive evaluation as many more hypotheses need to be evaluated to calculate the precision/recall curve. Calculating the 6D pose errors is more expensive than e.g. calculating the intersection over union of 2D bounding boxes (used to evaluate 2D object detection).
  2. The scores on the simpler 6D localization task are far from being saturated.

7.3 Discussion on the pose-error functions

Evaluating 6D object pose estimates is not straightforward. Only a fraction of an object’s surface may be visible in an image due to self-occlusion and occlusion by other objects. The visible part of the object may have multiple fits to the object’s 3D model. As a consequence, there may be (infinitely) many 6D poses consistent with the image. It is often difficult to explicitly enumerate all consistent poses.

The most widely used pose-error functions have been ADD/ADI [2,5], where the error is calculated as the average distance from vertices of the object model in the ground-truth pose to vertices of the model in the estimated pose. The distance is measured between corresponding vertices if all views of the object are distinguishable (ADD). Otherwise, for objects with indistinguishable views, the distance is measured between a given vertex and its nearest neighbor in 3D space, which may not necessarily be the corresponding vertex (ADI). ADI can yield un-intuitively low errors even for poses that are distinguishable. Objects evaluated with ADI therefore tend to yield low pose errors although the estimated poses might not be visually well aligned. Another limitation of ADD/ADI comes from a high dependence on the geometry of the object model and the sampling density of its surface – the average distance is dominated by higher-frequency surface parts, such as the thread of a fuse. The maximum distance used in MSSD and MSPD is less dependent on the surface sampling density.

MSSD and MSPD handle pose ambiguities due to global object symmetries. MSSD is suitable for robotic grasping and MSPD for augmented reality applications. However, because both are calculated over the entire model surface, misalignments of invisible parts are penalized. This may not be desirable for applications such as robotic manipulation with suction cups where only the alignment of the visible part is relevant. VSD is calculated only over the visible object part. It evaluates the alignment of the object shape but not of its color – this is because most of the object models currently included in BOP have baked shadows and reflections in their textures, which makes it difficult to robustly evaluate color alignment. As each of VSD, MSSD and MSPD is relevant for a different target application, we use all three for the evaluation in BOP.

8. Organizers

Tomáš Hodaň, Czech Technical University in Prague
Martin Sundermeyer, DLR German Aerospace Center
Eric Brachmann, Heidelberg University
Bertram Drost, MVTec
Frank Michel, Technical University Dresden
Jiří Matas, Czech Technical University in Prague
Carsten Rother, Heidelberg University

References

[1] Hodaň, Michel et al.: BOP: Benchmark for 6D Object Pose Estimation, ECCV 2018.

[2] Hodaň et al.: On Evaluation of 6D Object Pose Estimation, ECCVW 2016.

[3] Drost et al.: Introducing MVTec ITODD - A Dataset for 3D Object Recognition in Industry, ICCVW 2017.

[4] Brachmann et al.: Uncertainty-Driven 6D Pose Estimation of Objects and Scenes from a Single RGB Image, CVPR 2016.

[5] Hinterstoisser et al.: Model based training, detection and pose estimation of texture-less 3d objects in heavily cluttered scenes, ACCV 2012.

[6] Newcombe et al.: KinectFusion: Real-time dense surface mapping and tracking, ISMAR 2011.

[7] Hodaň et al.: Photorealistic Image Synthesis for Object Instance Detection, ICIP 2019.

[8] Denninger, Sundermeyer et al.: BlenderProc, arXiv 2019.