A simple baseline for model-free unseen object detection/segmentation with Fast Segment Anything (FastSAM) and DINOv2. This three-stage approach can work for any object without retraining:

Onboarding stage: For each object in the test dataset, we randomly select 100 reference images from the onboarding video(s) and crop the object from these images using the provided 2D bounding box (except for dynamic onboarding in the HOT3D dataset, where a bounding box is only provided for the first frame and thus only one reference image is used). Then we calculate the CLS-token descriptors of the crops using DINOv2. This process generates a set of reference descriptors of size "num_objects x 100 x C" for the testing dataset, where "num_objects" represents the number of test objects, and "C" denotes the descriptor size.

Proposal stage: We generate object proposals using FastSAM. Each proposal is defined by a binary mask and a 2D bounding box of the mask.

Matching stage: We calculate the CLS-token DINOv2 descriptors for the FastSAM proposals and compare them with the reference descriptors using cosine similarity. This process generates a similarity matrix of size "num_objects x 100". We then average the similarity scores over the 100 views to obtain a “ score" of the proposal with respect to each test object. Finally, we assign an object ID to each proposal by selecting the highest score using argmax.