Deep Learning for the Detection of Melt Ponds on Arctic Sea Ice

Introduction and Context

As we know today, the Arctic1 is warming up to four times faster than the rest of the Earth,2 leading to predictions that it could become seasonally ice-free by the middle of this century.3 The rapid warming in the Arctic could potentially disrupt global climate patterns, including alterations in the behaviour of  jet streams.4 Consequences for Arctic people and the shipping industry are already evident.5

Melt Ponds and their Impact on the Arctic Climate

Strong seasonal changes in the Arctic surface structure occur in summer, when temperatures rise close to or above freezing. The melting of sea ice and snow leads to the formation of ‘melt ponds’, pools of water that collect on areas of lower elevation (Figure 1). They begin to appear around the end of May, covering up to 60-80 percent of the sea ice area at the peak of melt, and then refreeze in August and September.6 Melt ponds have a strong influence on the Arctic energy budget. Due to their darker colouring, they absorb significantly more sunlight than the reflective sea ice.7 This causes the surrounding areas to warm up, leading to further melt and contributing to the overall rate of sea ice loss.8

Monitoring and understanding the dynamics of melt ponds is essential for predicting sea ice melt and for assessing the feedback mechanisms that contribute to the amplification of Arctic warming. By incorporating melt pond data into climate models, it is possible to improve the accuracy of predictions related to Arctic climate change and the broader implications for global climate patterns.9 In particular, the spatial extent of melt ponds is a crucial parameter for climate models, but has not yet been sufficiently integrated into models.10 The development of methods for retrieval and analysis of melt pond is an ongoing research area. 

Methods for Observing Melt Ponds

Accessing melt ponds is hampered by the remoteness of the Arctic Ocean. Remote sensing techniques such as satellites and helicopters allow researchers to observe the Arctic from a distance. The area covered by melt ponds can be inferred from images taken by these platforms. 

Satellites are a promising method in the long term because they can monitor large parts of the Arctic on a regular basis. However, to date, high-resolution satellite images have been limited due to cloud cover or satellite orbit constraints.11 The low resolution imagery obtained in these circumstances is too coarse to accurately identify melt ponds. 

To fill this gap, helicopter measurements are used12 Helicopter measurements can obtain high-resolution imagery for specific areas and have the advantage of flexibility because they can targetareas of interest.

At the sensor level, visual imagery (VIS) has been most commonly used.13 VIS captures the reflected amount of sunlight and indicates melt ponds by their blueish to dark gray colour (Figure 1).14 A drawback of VIS imaging is its dependence on daylight and its sensitivity to different lighting conditions. 

This article focuses on thermal infrared (TIR) imagery instead (Figure 2).15 TIR measures the emissivity of thermal radiation at wavelengths around 10μm. TIR can extend the knowledge gained from VIS retrievals by allowing researchers to ask new questions about the thermal properties of melt ponds, such as heat conductivity. Furthermore, TIR can be used in the absence of daylight. To date, only low-resolution TIR satellite imagery is available. Therefore, TIR data is gathered by helicopter. The gathered data is then analysed and separated according to surface types. 

How TIR Segmentation Works

Semantic segmentation is an image analysis technique used to classify each pixel into a predefined category. 

In VIS, segementation is often achieved through traditional image processing techniques such as thresholding,16 in which a pixel value threshold is set to differentiate areas of different surface types. Other techniques have used edge detection methods to partition objects from images.17

For TIR imagery, spatially and temporally varying temperatures place special demands on the segmentation algorithm used. Due to its salinity, ocean water has a freezing temperature of -1.8 degrees Celsius. This means that sea ice can be warmer or colder than the surrounding ocean in summer (Figure 2). Simple thresholding approaches are not useful because the surface types can not be distinguished by their pixel value. Furthermore, object boundaries in TIR are challenged by smooth temperature transitions or no transitions at all (Figure 3).

TIR segmentation thus requires a more general method that can consider multiple features such as shape and size simultaneously. Deep learning-based approaches have the potential to help with TIR segmentation because they can learn complex patterns from data. So far, only a few studies have applied deep learning to melt pond segmentation.18

Feature Extraction with Convolutional Neural Networks

Deep learning involves a variety of architectures that mimic the neural network structure of the human brain.19 In particular, convolutional neural networks (CNN) are designed for feature extraction from image data.20 They excel at automatically learning patterns at different levels of abstraction, e.g., they can capture local features and spatial context simultaneously. For example, a CNN can recognise that a circular shape alone does not necessarily indicate a melt pond, but must also be embedded in sea ice to differentiate it from an ice floe. This is done using so-called convolutional layers, which apply mathematical operations in the form of filters to the image dimensions.

Semantic Segmentation with U-net

To be suitable for the task of segmentation, a convolutional neural network (CNN) needs to be embedded in a more complex architecture that is able to map the learned features back to the pixel space. A famous architecture in this respect is U-net.21 U-net combines a CNN as a downscaling path with a decoder that upscales the feature representation and outputs a segmentation map (Figure 4).

U-net has achieved significant results in a variety of fields, including satellite image analysis,22 and is one of the most widely used architectures in remote sensing.23 U-net was therefore the model of choice for my work.

Training of Deep Learning Models

To be able to perform well on unseen images, deep learning models must be trained on a large amount of labeled data, typically consisting of thousands or millions of images. This training data needs to be representative, ideally including observations under a variety of conditions. This is challenging for the task of melt pond segmentation, where image acquisition is limited and manual labeling is time-consuming. In practice, several hours of work are required to annotate a single image. 

U-net has been shown to perform well even with relatively few training images.24 In addition, performance results can be improved using techniques such as augmentation and pre-training. Augmentation means transforming existing training data in advance to increase the variation of the dataset.25 This can be done by operations such as flipping or rotating input images. Pre-trained models that have already learned general features from a much larger dataset can be adapted to the task as well.26

Experiments and Results

To test U-net’s usefulness for detecting melt ponds using TIR, I labeled ten TIR images taken during a helicopter flight during the PS131 ATWAICE 2022 campaign.27 I used a U-net pre-trained on ImageNet, one of the largest datasets available for machine learning.28 I optimised some of the configurable hyperparameters, such as the loss function, the patch size and the pre-training strategy, over several training setups and tested different augmentation techniques.29

The results are shown in Figure 5. While some images can be predicted relatively well, this is not the case everywhere. In particular, images taken on a different flight than the training data were segmented incorrectly. 

Conclusion

Achieving accurate segmentation of TIR images would open avenues for better understanding the characteristics of Arctic sea ice during the summer. Deep learning methods can help with the challenging task of TIR segmentation, as shown by its successful predictions in the second column of Figure 5. So far, the generalisability of the model is hampered due to limited training data. The development of an operational method remains an ongoing research area, with a particular need to label more representative training data from different flights and areas.