The main belt is believed to have originally contained an Earth mass or more of material, enough to allow the asteroids to accrete on relatively short timescales. The present-day main belt, however, only contains ̃5×10 Earth masses. Numerical simulations suggest that this mass loss can be explained by the dynamical depletion of main belt material via gravitational perturbations from planetary embryos and a newly-formed Jupiter. To explore this scenario, we combined dynamical results from Petit et al. [Petit, J. Morbidelli, A., Chambers, J., 2001. The primordial excitation and clearing of the asteroid belt. Icarus 153, 338-347] with a collisional evolution code capable of tracking how the main belt undergoes comminution and dynamical depletion over 4.6 Gyr [Bottke, W.F., Durda, D., Nesvorny, D., Jedicke, R., Morbidelli, A., Vokrouhlický, D., Levison, H., 2005. The fossilized size distribution of the main asteroid belt. Icarus 175, 111-140]. Our results were constrained by the main belt's size-frequency distribution, the number of asteroid families produced by disruption events from diameter D>100 km parent bodies over the last 3-4 Gyr, the presence of a single large impact crater on Vesta's intact basaltic crust, and the relatively constant lunar and terrestrial impactor flux over the last 3 Gyr. We used our model to set limits on the initial size of the main belt as well as Jupiter's formation time. We find the most likely formation time for Jupiter was 3.3±2.6 Myr after the onset of fragmentation in the main belt. These results are consistent with the estimated mean disk lifetime of 3 Myr predicted by Haisch et al. [Haisch, K.E., Lada, E.A., Lada, C.J., 2001. Disk frequencies and lifetimes in young clusters. Astrophys. J. 553, L153-L156]. The post-accretion main belt population, in the form of diameter D≲1000 km planetesimals, was likely to have been 160±40 times the current main belt's mass. This corresponds to 0.06-0.1 Earth masses, only a small fraction of the total mass thought to have existed in the main belt zone during planet formation. The remaining mass was most likely taken up by planetary embryos formed in the same region. Our results suggest that numerous D>200 km planetesimals disrupted early in Solar System history, but only a small fraction of their fragments survived the dynamical depletion event described above. We believe this may explain the limited presence of iron-rich M-type, olivine-rich A-type, and non-Vesta V-type asteroids in the main belt today. The collisional lifetimes determined for main belt asteroids agree with the cosmic ray exposure ages of stony meteorites and are consistent with the limited collisional evolution detected among large Koronis family members. Using the same model, we investigated the near-Earth object (NEO) population. We show the shape of the NEO size distribution is a reflection of the main belt population, with main belt asteroids driven to resonances by Yarkovsky thermal forces. We used our model of the NEO population over the last 3 Gyr, which is consistent with the current population determined by telescopic and satellite data, to explore whether the majority of small craters ( D<0.1-1 km) formed on Mercury, the Moon, and Mars were produced by primary impacts or by secondary impacts generated by ejecta from large craters. Our results suggest that most small craters formed on these worlds were a by-product of secondary rather than primary impacts.