Structural diversity, degree of freedom, and reliability in climate model ensembles

Due to our lack of understanding of the climate system and limitations of computational power, climate models are far from perfect. The different models do, however, span a considerable range of output which leads to the possibility of making probabilistic pre-dictions of the future based on the models. How best to integrate ensembles of models into a probabilistic calculation is still a matter of debate. One approach to generating probabilistic future predictions is to implement a weighting procedure based on the per-formance of the present day climate simulation. One of the prerequisites for implemen-tation of such a method is that the ensemble employed should initially be broad enough to include the truth. Therefore, understanding the characteristics of the ensembles that have already been generated is an important step in this process. In the present study, we investigate the reliability of various types of climate model ensembles using several novel methodologies for the model evaluation. Here, we focus on assessing reliability against present-day climatology. One of the methodologies is the "rank histogram approach" [1,2], which is often used for the evaluations of numerical weather forecasts. In addition, we also perform the analyses of effective degrees of freedom (EDoFs) of the climate variables, and the distances between the observation and climate model ensemble members. We investigate the performance of the newest generation multi-model ensemble (MME) from the Coupled Model Intercomparison Project (CMIP5). We compare the ensemble to the previous generation models (CMIP3) as well as several single model ensembles (SMEs), which are constructed by varying components of single models. These SMEs range from ensembles where pa-rameter uncertainties are sampled (perturbed physics ensembles) through to an ensem-ble where a number of the physical schemes are switched (multi-physics ensemble). Through these analyses, we find that the features of the CMIP5 rank histograms, of general reliability on broad scales, are consistent with those of CMIP3, suggesting a similar level of performance for present-day climatology. The spread of MMEs tends towards being "too wide" rather than "too narrow". In general, the SMEs examined tend towards insufficient dispersion and the rank histogram analysis identifies them as being statistically distinguishable from many of the observations. The EDoFs of the MMEs are generally greater than those of SMEs, suggesting that structural changes lead to a characteristically richer range of model behaviours than parametric/physical-scheme-switching ensembles. For distance measures, the observations and models ensemble members are similarly spaced from each other for MMEs, whereas for the SMEs, the observations are generally well outside the ensemble. We suggest that multi-model en-sembles should represent an important component of uncertainty analysis. References: [1] Annan JD, Hargreaves JC (2010) Reliability of the CMIP3 ensemble, Geophys. Res. Lett., 37, L02703, doi:10.1029/2009GL041994. [2] Yokohata T, Annan JD, Collins M, Jackson CS, Tobis M, Hargreaves JC (2011) Re-liability of multi-model and structurally different single-model ensembles, Clim Dyn, DOI: 10.1007/s00382-011-1203-1, 2011