Recently Dantas, Oliveira and Stilck [J. Stat. Mech. (2007) P08009] studied how the one-dimensional diffusive contact process crosses over from the critical behavior of directed percolation to an effective mean field behaviour when the diffusion rate is sent to infinity. They showed that this crossover can be described in terms of a crossover exponent $\phi$, finding the boundaries 3 <= $\phi$ <= 4 in one spatial dimension. In the present work we refine and extend this result up to four spatial dimensions by a field-theoretic calculation and extensive numerical simulations.