Submitted to: BOP Challenge 2023

Training data: 3D point clouds of indoor scenes from the 3DMatch dataset [A]

Onboarding data: No need for onboarding

Used 3D models: Default point clouds provided in: "models_reconst" for T-LESS, "models_eval" for ITODD, "models" for the other datasets.

Notes: PoZe performs pose estimation of unseen objects through zero-shot learning. PoZe takes as input a colored 3D point cloud that represents the object and an RGB-D image capturing the scene. PoZe consists of five modules:

• 2D object segmentation: We segment the region that the object occupies in the RGB image. We use the segmentation masks predicted by CNOS [B].
• 3D lifting: We crop the input image around the segmentation mask and back-project the cropped scene in the 3D space by using the camera intrinsic parameters.
• Feature extraction: We extract point-wise features from the point clouds of the object and the cropped scene. We use a frozen GeDi [C] model trained on 3DMatch for the point cloud registration task. We extract features at three different scales.
• Pose estimation: We estimate the 6D pose of the object with respect to the cropped scene using feature-matching RANSAC on the features extracted at each scale. Thereafter, we select the best estimated pose according to the number of RANSAC inliers.
• Pose refinement: We refine the pose estimated by the previous module by using the ICP algorithm.

[A] Zeng et al.: 3DMatch: Learning the matching of local 3D geometry in range scans, CVPR 2017
[B] Nguyen et al.: CNOS: A Strong Baseline for CAD-based Novel Object Segmentation, arXiv 2023
[C] Poiesi et al.: Learning general and distinctive 3D local deep descriptors for point cloud registration, IEEE PAMI 2023