The Damocloids are objects thought, on dynamical grounds, to be inactive Halley-family and long-period comets. We present optical measurements of 12 such objects, finding that their mean Kron-Cousins colors are B-V=0.79+/-0.01, V-R=0.48+/-0.01, and R-I=0.48+/-0.01. The normalized reflectivity spectra are generally linear, with a mean gradient S'=11.9%+/-1.0% per 1000 Å. The latter is consistent with the mean S'=11.6%+/-2.3% per 1000 Å measured for the nuclei of (short-period) Jupiter-family comets, a surprising result given the expected very different formation locations and dynamical histories of these two types of body. The Damocloids are devoid of the ultrared matter (with S'>=25% per 1000 Å) that is present on many Kuiper belt objects and Centaurs, and the mean colors of the Damocloids are inconsistent with those of the Kuiper belt objects (S'=21.1%+/-1.4% per 1000 Å). The data suggest that the ultrared matter, widely thought to consist of a complex organic compound processed by prolonged exposure to cosmic rays, cannot survive long in the inner solar system. Timescales for ejection or burial of ultrared matter on the nuclei of both Jupiter-family comets and Damocloids are short. Such material may also be chemically unstable to the higher temperatures experienced in the inner planetary region.