Submitted to: BOP Challenge 2024

MUSE: Model-agnostic Unseen 2D Object Recognition via 3D-aware Similarity of Multi-Embeddings We present MUSE, a training-free and model-agnostic framework for unseen 2D object recognition, leveraging 3D-aware similarity computed from multi-embedding descriptors.

Specifically, MUSE integrates class-level and patch-level embeddings into a novel similarity metric, and introduces the Integrated von Mises-Fisher (I-vMF) similarity, which applies the von Mises-Fisher (vMF) distribution to weigh the contributions of 3D template views. This weighting reflects the assumption that high similarity scores are concentrated around the correct template view on the viewing sphere.

To further enhance reliability, we propose Confidence-Assisted Similarity (CAS), which modulates the I-vMF similarity using the uncertainty estimate of the vision model, giving more influence to confident predictions.

As our approach relies solely on similarity computations over feature embeddings, MUSE is fully model-agnostic and can be integrated with any vision backbone without fine-tuning.

In our implementation, we use Grounding DINO and SAM2 to extract detection proposals, and adopt DINOv2-Large as the feature encoder for computing multi-level similarity.

Authors – Temporary Anonymous