Evaluation of the sensitivity of learning rate on the training of neural network hydrologic models

The artificial neural networks (ANNs) have become the most exploited tool for hydrologic modeling in recent years. However, despite an explosive growth of ANN applications for hydrologic modeling observed over the years, the ANN training remains a trial and error procedure. Such trial and error procedures are often inefficient in terms of computational effort and may not be able to ensure optimal solutions. There are no standard guidelines that can be employed uniformly for the training of ANN models. Determination of an optimal ANN architecture has remained a trial and error procedure over the years. This may be because each problem of hydrologic modeling is unique, and there are many factors that dictate the training process such as learning rate, momentum factor, not to mention computationally intensive methodologies that must be followed to avoid under-training and/or over-training of the ANN models. Therefore, there appears to be a strong need to examine the training process closely and determine the sensitivity of various training parameters on the optimal training that is essential to determine the best ANN architecture for a given data set. This paper investigates the sensitivity of the learning rate on the training of ANN hydrologic models. The back-propagation training method in batch mode was used for this purpose. The rainfall and flow data from Kentucky River basin, USA were first used to develop a suitable ANN hydrologic model. The ANN model selected for sensitivity analysis consists of eight neurons in the input layer representing various rainfall and flow values in the past and upstream locations; five neurons in the hidden layer; and a single neuron in the output layer representing the flow at time t being modeled. The learning rate was varied from 0.0001 to 0.05 and the constant stopping criteria were achieved for each learning rate. The stopping criteria consisted of 10,000 iterations or SSE=0.0005. The number of iterations needed to reach the stopping criteria were noted for each learning rate. The results obtained in this study indicate that the complex dynamics inherent in the input and output data correspond to an optimal range of learning rate that must be determined to achieve best results. It was found that very small learning rates were not able to achieve the desired convergence in terms of SSE and hence not able to train the ANN hydrologic models properly. As the learning rate is increased, the number of iterations needed to achieve the acceptable error first decrease and then increase after an optimal plateau. The training process becomes unstable at moderately high learning rates and saturates leading to no-training at very high values of learning rates. The results obtained in this study point towards the need of developing certain guidelines that can be followed in determining the optimal ANN architecture for a given hydrologic modeling problem with least effort during training.