We use a Babcock-Leighton-type of dynamo with the poloidal source based as closely as possible on the observations to model the solar cycle irregularities during the cycles 12-20. The nonlinearities in the poloidal field generation, i.e., the cycle dependence of the sunspot group tilt angles and the latitudes are included. The convective pumping dominates the transport of the surface poloidal field to the tachocline. Our results show that the modeled polar fields have a good correlation with the observed next cycle strength and the build-up of the toroidal flux at the tachocline is strongly correlated with the observed cycle strength as well. Both correlation coefficients are above 0.8. The success of the model indicates that they are the nonlinearities in the poloidal field generation which cause the solar cycle irregularities.