The geophysical history of Mercury is constrained by its crater record, its tectonics, and its magnetic field. Standard thermal models based on these constraints lead to inconsistent results unless Mercury was bombarded in its early history by an additional population of impactors. It has been suggested that the putative vulcanoid population, a belt of asteroid-like bodies residing on stable orbits inside the orbit of Mercury, may be the source of these impactors. Previous work has shown that collisions among vulcanoids are sufficient to deplete the stable vulcanoid zone of most multikilometer bodies within a billion years or so. In this paper, we examine whether the Yarkovsky effect, a thermal radiation force which forces asteroids to undergo semimajor axis drift as a function of their spin, orbit, and material properties, is strong enough to deplete the remaining material from the vulcanoid zone. Our results show that most kilometer-sized bodies escape into unstable orbits within a few billion years. We predict that the contemporary vulcanoid population, if it exists at all, may be limited to 300-900 bodies larger than 1 km in diameter.