Many neutron stars are observed to be moving with spatial velocities, in excess of 500 km s-1. A number of mechanisms have been proposed to give neutron stars these high velocities. One of the leading classes of models proposed invokes asymmetries in the core of a massive star just prior to collapse. These asymmetries grow during the collapse, causing the resultant supernova to be also asymmetric. As the ejecta is launched, it pushes off (or ``kicks'') the newly formed neutron star. This Letter presents the first three-dimensional supernova simulations of this process. The ejecta is not the only matter that kicks the newly formed neutron star. Neutrinos also carry away momentum and the asymmetric collapse also leads to asymmetries in the neutrinos. However, the neutrino asymmetries tend to damp out the neutron star motions, and even the most extreme asymmetric collapses presented here do not produce final neutron star velocities above 200 km s-1.