overview
GVSL in 3D Medical Image Self-supervised Pre-training

[Accepted by CVPR 2023]

  • 1Southeast University
  • 2Nanjing University of Aeronautics and Astronautics
  • 3University of Rennes 1
  • 4Western University
  • 5Case Western Reserve University

Overview

overview

Learning inter-image similarity is crucial for 3D medical images self-supervised pre-training, due to their sharing of numerous same semantic regions. However, the lack of the semantic prior in metrics and the semantic-independent variation in 3D medical images make it challenging to get a reliable measurement for the inter-image similarity, hindering the learning of consistent representation for the same semantics. We investigate the challenging problem of this task, i.e., learning a consistent representation between images for a clustering effect of the same semantic features. We propose a novel visual similarity learning paradigm, Geometric Visual Similarity Learning, which embeds the prior of topological invariance into the measurement of the inter-image similarity for consistent representation of semantic regions.

Video

(Waiting...)

Highlights

  • Advance the learning of inter-image similarity in 3D medical image self-supervised pre-training and push the representability of same visual semantics between images.
  • A powerful and novel pretext learning paradigm, Geometric Visual Similarity Learning, is proposed to embeds the prior of topological invariance into the measurement for reliable learning.
  • A reliable and novel projection head, Z-Matching head, is proposed for simultaneously powerful global and local representation in learning geometric matching-based pre-training.
  • Experiments demonstrates our learning of inter-image similarity yields more powerful inner-scene, inter-scene, and global-local transferring ability on four challenging 3D medical image tasks.

Why learning inter-image similarity is crucial for 3D medical image SSP?

overview
  • Same category 3D medical images share numerous same semantic regions due to the consistent human anatomies and the complete spatial information in 3D vision, having large inter-image similarity.
  • Natural images have large semantic difference between images whose inter-image similarity is weak.

    • Challenge

    It is challenging to measure a reliable inter-image similarity.
  • Large appearance similarity between different semantic regions: the Myo and RA regions in images A and B.
  • Appearance dissimilarity between same semantic regions: RA regions are different in images B and C.
    • overview

    Motivation

    overview
    The topological invariance of the visual semantics between the 3D medical images provides a motivation to discover their inter-image correspondence.

    The framework of our GVSL

    overview

  • Our GVSL learns the GM from the representation of the semantics in images, thus driving the learning of inter-image similarity via the gradient in backpropagation.
  • Our Z-Matching head learns affine and deformable matchings simultaneously for powerful global and local representations.
  • For efficient learning, it also takes a fundamental pretext task, the self-restoration, for a basic representation of semantics, thus giving a warm-up for GM learning.

    • Intuitions on GVSL’s behavior

    The prior of topological invariance in GM embeds a topology manifold into the metric, thus bringing an efficient measurement for inter-image similarity and guiding the clustering effect of the same semantic features.
    overview

    BibTeX

    If you find our project or pre-trained parameters useful in your research, please cite: 
    @InProceedings{He_2023_CVPR,
    author    = {Yuting He, Guanyu Yang, Rongjun Ge, Yang Chen, Jean-Louis Coatrieux, Boyu Wang, Shuo Li},
    title     = {Geometric Visual Similarity Learning in 3D Medical Image Self-supervised Pre-training},
    booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
    month     = {June},
    year      = {2023},
    pages     = {}
}

    Acknowledgments

    The website template was borrowed from Chaoyi Zhang.