Speeding up the Hyperparameter Optimization of Deep Convolutional Neural Networks
Abstract
Most learning algorithms require the practitioner to manually set the values of many hyperparameters before the learning process can begin. However, with modern algorithms, the evaluation of a given hyperparameter setting can take a considerable amount of time and the search space is often very highdimensional. We suggest using a lowerdimensional representation of the original data to quickly identify promising areas in the hyperparameter space. This information can then be used to initialize the optimization algorithm for the original, higherdimensional data. We compare this approach with the standard procedure of optimizing the hyperparameters only on the original input. We perform experiments with various stateoftheart hyperparameter optimization algorithms such as random search, the tree of parzen estimators (TPEs), sequential modelbased algorithm configuration (SMAC), and a genetic algorithm (GA). Our experiments indicate that it is possible to speed up the optimization process by using lowerdimensional data representations at the beginning, while increasing the dimensionality of the input later in the optimization process. This is independent of the underlying optimization procedure, making the approach promising for many existing hyperparameter optimization algorithms.
 Publication:

arXiv eprints
 Pub Date:
 July 2018
 arXiv:
 arXiv:1807.07362
 Bibcode:
 2018arXiv180707362H
 Keywords:

 Computer Science  Neural and Evolutionary Computing;
 Computer Science  Machine Learning
 EPrint:
 15 pages, published in the International Journal of Computational Intelligence and Applications