Evaluating the Spatial Success of Landslide Prediction Models using Visual Contingency Tables

Accurate prediction of areas susceptible to and impacted by landslides can be challenging. Commonly used Receiver-Operator Characteristic (ROC) analysis is a method for evaluation of predictive success of landslide models; however, for prediction of spatial outcomes such as landslide susceptibility, ROC curves are limited because they do not provide insight into spatial patterns of predictive success. To highlight spatial patterns, we supplement ROC analysis with visual contingency tables (VCT), map representations of the matrix formed by a contingency table. VCT representation provides a visualization of the four categories of predictive success and thereby allows closer examination of the information contained in a single point on a ROC plot.

We applied VCT analysis to landslides and long-runout debris flows mapped in four study areas in Puerto Rico affected by Hurricane Maria in 2017. Our analysis was applied to the entire landslide affected area (source, transport, and deposition zones), and an approximation of source-zone only. We first determined landslide susceptibility predicted in topographic terrains: 1) steep slopes, 2) concave terrain, 3) headwall swales, and 4) steep slopes adjacent to the drainage network. The visualization provided by VCT revealed several spatial relations that were not apparent in ROC analysis. For example, some source zones for debris-flow generating landslides initiated in concave topography adjacent to drainage networks, hence were predicted with headwall swales. However, others initiated on planar topography adjacent to headwall swales. This information allowed optimal adjustment of a curvature threshold and thereby improved predictive success of source zones for long-runout debris flows. VCT allowed us to identify cut-slope failures and stream bank collapse better than metrics such as distance from roads or streams. VCT also identified outlier cases not predicted well by the susceptibility models, such as landslides with atypically low slopes. This identification allowed focused evaluation to identify inaccuracies in source-zone approximation. VCT analysis provides a useful supplement to standard ROC analysis, enabling direct insights into the spatial patterns governing the success of a landslide prediction model.