Method: CNOS (FastSAM)

User nvnguyen
Publication https://arxiv.org/pdf/2307.11067.pdf
Implementation https://github.com/nv-nguyen/cnos
Training image modalities None
Test image modalities RGB
Description

A simple baseline for unseen object detection/segmentation with Fast Segment Anything (FastSAM) and DINOv2. This three-stage approach can work for any object without retraining:

Onboarding stage: For each object in the test dataset, we select 42 reference images from "PBR-BlenderProc4BOP" training images and crop the object from these images using the ground-truth modal 2D bounding box.Then we calculate the CLS-token descriptors of the crops using DINOv2. This process generates a set of reference descriptors of size "num_objects x 42 x C" for the testing dataset, where "num_objects" represents the number of test objects, and "C" denotes the descriptor size.

Proposal stage: We generate object proposals using FastSAM (as in the FastSAM paper). Each proposal is defined by a binary mask and a 2D bounding box of the mask.

Matching stage: We calculate the CLS-token DINOv2 descriptors for the FastSAM proposals and compare them with the reference descriptors using cosine similarity. This process generates a similarity matrix of size "num_objects x 42". We then average the similarity scores over the 42 views to obtain a “ score" of the proposal with respect to each test object. Finally, we assign an object ID to each proposal by selecting the highest score using argmax.

Important: The method predicts modal masks (covering just the visible object parts). The 2D bounding boxes are not explicitly predicted but calculated from the modal masks. The predicted boxes are therefore modal while the GT boxes used in the BOP evaluation are amodal (covering also the invisible object part), which yields lower detection scores.

Computer specifications V100 16GB

Public submissions

Date Submission name Dataset
2023-08-02 12:44 FastSAM LM-O
2023-08-02 12:44 FastSAM IC-BIN
2023-08-02 12:44 FastSAM T-LESS
2023-08-02 12:45 FastSAM ITODD
2023-08-02 12:45 FastSAM HB
2023-08-02 12:45 FastSAM YCB-V
2023-08-02 12:46 FastSAM TUD-L
2023-08-02 12:50 FastSAM T-LESS
2023-08-02 12:51 FastSAM TUD-L
2023-08-02 12:51 FastSAM LM-O
2023-08-02 12:52 FastSAM IC-BIN
2023-08-02 12:52 FastSAM ITODD
2023-08-02 12:52 FastSAM HB
2023-08-02 12:52 FastSAM YCB-V