3DPoV

Abstract

Visual foundation models have achieved remarkable progress in scale and versatility, yet understanding the 3D world remains a fundamental challenge. While 2D images contain cues about 3D structure that humans readily interpret, deep models often fail to exploit them, underperforming on tasks such as multiview semantic consistency – crucial for applications including robotics and autonomous driving. We propose a self-supervised approach to enhance the 3D understanding of vision foundation models by (i) introducing a temporal nearest-neighbor consistency loss that finds corresponding points across video frames and enforces consistency between their nearest neighbors, (ii) incorporating reference-guided ordering that requires patch-level features to be not only expressive but also consistently aligned, and (iii) constructing a mixture of video datasets tailored to these objectives, thereby leveraging rich 3D information. Our method, 3DPoV, achieves state-of-the-art performance in keypoint matching under viewpoint variation, as well as in depth and surface normal estimation, and consistently improves a diverse set of backbones, including DINOv3.

Method

3DPoV uses a teacher–student architecture where the student processes video frames independently and the teacher provides a stable, EMA-updated anchor frame for supervision. Rather than matching features directly, we align the relative similarity structure of tracked patches over time: for each frame we compute similarity rankings against a shared bank of reference patches and use differentiable sorting to obtain soft permutation matrices, training the student to match the teacher's anchor-frame permutations. This yields viewpoint-invariant descriptors that stay consistent under motion, occlusion, and lighting changes — without relying on crops or masks.

Fine-tuning blends three complementary video sources: CO3D (long object-centric sequences with large viewpoint shifts), DL3DV (diverse dynamic scenes and spatial layouts), and YouTube-VOS (unconstrained motion, occlusions, and real-world camera trajectories) — together spanning object-, scene-, and natural-video variability, with only ~20 hours of training on A100 GPUs.

Results

Keypoint Matching — SPair-71k

Recall across viewpoint difficulty bins (S: same / M: moderate / L: large). 3DPoV improves consistently, including under large viewpoint shifts.

Keypoint Matching — Navi

Recall binned by relative camera rotation. 3DPoV improves uniformly across all viewpoint regimes.

Depth Estimation — Navi

Surface Normal Estimation — Navi

Robustness to Occlusion

SPair-71k recall with synthetic occlusions applied. 3DPoV improves consistently across all difficulty levels and both occlusion settings.

Robustness to Lighting

Navi keypoint recall binned by viewpoint across three lighting conditions.

BibTeX

@inproceedings{simion2026_3dpov,
  title     = {3DPoV: Improving 3D Understanding via Patch Ordering on Videos},
  author    = {Simion, Ioana and Salehi, Mohammadreza and Venkataramanan, Shashanka and Snoek, Cees G. M. and Asano, Yuki M.},
  booktitle = {Proceedings of the 43rd International Conference on Machine Learning (ICML)},
  series    = {PMLR},
  volume    = {306},
  year      = {2026}
}