SLICE: Self-Supervised Learning for Images of a Changing Earth

The frequency and magnitude of Earth remote sensing data often necessitates data analysis methodologies such as machine and deep learning to discover & classify occurrences of phenomena, segment the image, uncover trends, and produce scientific & actionable insights. However, most of these imagery datasets are not labeled at scale due to the need for specialized domain knowledge. Due to the lack of labeled data, approaches such as unsupervised or self-supervised learning (SSL) are needed to achieve accuracy even when only a small number of labels are available.

The state-of-the-art in SSL techniques is advancing rapidly, with multiple algorithms recently published: SimCLRv2, DINO, ViT (Vision Transformers), EsViT and MViT. SimCLRv2 from Google has already been applied by ourselves to Mars imagery and by others to Earth imagery, but there is an even larger opportunity to exploit the latest Contrastive (loss) pre-trained network architectures and transfer learning at scale for applications of Earth image classification, smart segmentation, phenomena detection, and pattern evolution.

Under a NASA AIST grant, we are applying, tuning and comparing a set of SSL techniques on Earth imagery problems to characterize accuracy, training time, model size, inference time, etc. Target problems include patterns in A-Train atmospheric or surface variables (e.g. brightness temperature, aerosols), sea surface and eddy studies using surface temperature anomaly (SSTA) and sea surface height (SSH), land cover classification from Synthetic Aperture Radar (SAR) datasets (for NiSAR), etc. For the first application, we are detecting mesoscale and sub-mesoscale ocean eddies using SST, SSH and SAR data, and characterizing key physical properties such as heat flux. Horizontal and vertical heat transport is a key driver of ocean evolution.

The presentation will include an overview of SSL and ViT, use of contrastive loss and other loss functions, classification and segmentation results, and first results from the ocean eddies work. Figure 1 depicts the three stage highly-parallel SSL framework we are developing: unsupervised training using a large set of unlabeled images, supervised finetuning uisng a small set of labeled images, and ensemble predictions from the fine-tuned model.