In this study we attempt to model, with numerical simulations, the scenario for the origin of the late heavy bombardment (LHB) proposed in Chambers (2007, Icarus, 189, 386). Chambers suggested that the orbit of a fictitious sub-Mars sized embryo outside Mars became unstable and crossed a part of the asteroid belt until it was removed through ejection or collision. Chambers demonstrated that this kind of evolution can occur at the LHB time, but he did not check whether enough material could be liberated from the belt to be consistent with the magnitude of the LHB. We do that here. In addition to the asteroid belt, we also consider putative belts of small objects located between the orbits of the Earth and Venus (EV belt) or between the Earth and Mars (EM belt). We find that delivering enough material to the Moon from the asteroid belt requires at least a 5 lunar mass embryo crossing the entire asteroid belt up to 3.3 AU for longer than 300 Myr on a low-inclination orbit. We do not witness this to occur in over a hundred simulations. An alternative approach, which relies only on decimating the inner belt ( < 2.5 AU), requires that the inner belt originally contained 4 to 13 times more mass than the original outer belt. Regarding the putative asteroid belts in between the terrestrial planets, this scenario could be consistent with the LHB if EV belt contained a substantial population of small bodies, with the density of objects in orbital space larger than that of the current main asteroid belt, while the EM belt was as good as empty.