Effects of experimental protocol (baseline climatology, validation data selection and map- comparison methods) on the evaluation of global vegetation model accuracy: a comparison of simulated and observed vegetation patterns for Asia

Prognostic vegetation models have been widely used to study the interactions between environmental change and biological systems. This study examines the sensitivity of vegetation simulations to: (i) input mean climatologies of different length, (ii) the choice of observed data for evaluating the model results, and (iii) the method used to compare simulated and observed vegetation. We use vegetation simulated for Asia by the equilibrium vegetation model BIOME4 as a typical example. BIOME4 was run under 19 climatologies of different lengths derived from the CRU TS 2.0 data set. The 19 climate-driven scenarios extend backward in time from December of 1992 for 2, 5, 10, ~{!-~}, 90 years respectively. The year 1992 was chosen as the common end point for the climatologies because three global land- and tree-cover data sets, i.e. the Global Land Cover Characteristic (GLCC) data, the Global Land Cover Facility (GLCF) data, and the Ramankutty and Foley global Potential Natural Vegetation data (PNV) used to evaluate the model results were developed from remotely sensed images taken over the period of April 1992 to March 1993. The Kappa statistic, Fuzzy Kappa and a newly developed map comparison method, the Nomad index, were used to quantify the agreement between the BIOME4 biomes simulated under each climate scenario and those derived from three land- and tree-cover data sets. The results indicate that 30- and 35-year mean climatologies for the time period immediately preceding the origin date of the observed data produce the most consistent and accurate vegetation simulations when compared with all three observed data sets. The study also indicates that the BIOME4-simulated vegetation for Asia more closely matches the PNV data than the other two observed vegetation data sets. Given the same observed data, different methods of map comparisons can lead to differing judgment on the accuracy of a vegetation model to simulate terrestrial vegetation. The results of this analysis can assist model users in designing experimental protocols for simulating vegetation.