We simulate the inner 100 pc of the Milky Way to study the formation and evolution of the population of star clusters and intermediate-mass black holes (IMBHs). For this study we perform extensive direct N-body simulations of the star clusters that reside in the bulge, and of the inner few tenth of parsecs of the supermassive black hole in the Galactic center. In our N-body simulations the dynamical friction of the star cluster in the tidal field of the bulge are taken into account via semianalytic solutions. The N-body calculations are used to calibrate a semianalytic model of the formation and evolution of the bulge. We find that ~10% of the clusters born within ~100 pc of the Galactic center undergo core collapse during their inward migration and form IMBHs via runaway stellar merging. After the clusters dissolve, these IMBHs continue their inward drift, carrying a few of the most massive stars with them. We predict that a region within ~10 pc of the supermassive black hole (SMBH) is populated by ~50 IMBHs of ~1000 Msolar. Several of these are still expected to be accompanied by some of the most massive stars from the star cluster. We also find that within a few milliparsecs of the SMBH there is a steady population of several IMBHs. This population drives the merger rate between IMBHs and the SMBH at a rate of about one per 10 Myr, sufficient to build the accumulated majority of mass of the SMBH. Mergers of IMBHs with SMBHs throughout the universe are detectable by LISA at a rate of about two per week.