On the versatility of popular and recently proposed supervised evaluation metrics for segmentation quality of remotely sensed images: An experimental case study of building extraction
Abstract
Geographic Object-Based Image Analysis (GEOBIA) is a popular classification approach for high spatial resolution (HSR) images in the remote sensing (RS) community. It has been extensively reported that the final classification accuracy of GEOBIA is highly dependent on the quality of image segmentation. Different supervised, unsupervised, and post-classification quality assessment approaches have been proposed to evaluate whether a segmentation result has acceptable quality. Although there have recently been some theoretical review papers on supervised approaches in the context of RS, there have not been any comprehensively experimental tests on comparing these approaches for nearly a decade. Considering this gap, we experimentally compared different recently proposed and popular segmentation evaluation metrics to help the users narrow down the metrics that can be reliably used to measure the quality of a segmentation in a supervised manner. We assessed 21 distinguishable under-segmentation (US), over-segmentation (OS), and combined (UO) metrics in a two-step experimental procedure. The experiments showed that the OS metrics (including Recall, OSeg, NSR, and OSE) and US metrics (Precision, USeg, PSE, USE, and PI) each suffered from some problems including, but not limited to, unsuitable weighting (e.g. NSR), not enough sensitivity to segmentation variations (e.g. OSE/USE), uncertainty/unreliability in some cases (e.g. USeg/PSE), etc. Although none of the UO metrics was able to find the most optimal segmentation found with visual inspection, the UO metrics including F_measure, QR, and SEI could identify an optimal segmentation that was still of high quality and was very similar to the one visually found. It was also found that these three UO metrics performed very similar to each other, and that OI2 and AFI also led to similar results although AFI's unbounded range of values is a source of confusion about what is the most reliable way to choose optimal segmentations in this approach.
- Publication:
-
ISPRS Journal of Photogrammetry and Remote Sensing
- Pub Date:
- February 2020
- DOI:
- 10.1016/j.isprsjprs.2020.01.002
- Bibcode:
- 2020JPRS..160..275J
- Keywords:
-
- Geographic object-based image analysis (GOBIA);
- OBIA;
- Image segmentation;
- Segmentation quality;
- Supervised evaluation