The known irregular satellites of the giant planets are dormant comet-like objects that reside on stable prograde and retrograde orbits in a realm where planetary perturbations are only slightly larger than solar ones. Their size distributions and total numbers are surprisingly comparable to one another, with the observed populations at Jupiter, Saturn, and Uranus having remarkably shallow power-law slopes for objects larger than 8-10 km in diameter. Recent modeling work indicates that they may have been dynamically captured during a violent reshuffling event of the giant planets ~3.9 billion years ago that led to the clearing of an enormous, 35 M ⊕ disk of comet-like objects (i.e., the Nice model). Multiple close encounters between the giant planets at this time allowed some scattered comets near the encounters to be captured via three-body reactions. This implies the irregular satellites should be closely related to other dormant comet-like populations that presumably were produced at the same time from the same disk of objects (e.g., Trojan asteroids, Kuiper Belt, scattered disk). A critical problem with this idea, however, is that the size distribution of the Trojan asteroids and other related populations do not look at all like the irregular satellites. Here we use numerical codes to investigate whether collisional evolution between the irregular satellites over the last ~3.9 Gyr is sufficient to explain this difference. Starting with Trojan asteroid-like size distributions and testing a range of physical properties, we found that our model irregular satellite populations literally self-destruct over hundreds of Myr and lose ~99% of their starting mass. The survivors evolve to a low-mass size distribution similar to those observed, where they stay in steady state for billions of years. This explains why the different giant planet populations look like one another and provides more evidence that the Nice model may be viable. Our work also indicates that collisions produce ~0.001 lunar masses of dark dust at each giant planet, and that non-gravitational forces should drive most of it onto the outermost regular satellites. We argue that this scenario most easily explains the ubiquitous veneer of dark carbonaceous chondrite-like material seen on many prominent outer planet satellites (e.g., Callisto, Titan, Iapetus, Oberon, and Titania). Our model runs also provide strong indications that the irregular satellites were an important, perhaps even dominant, source of craters for many outer planet satellites.