The ratio of the mass of interstellar dust to the total mass of metals (the dust-to-metals/DTM ratio) tends to increase with metallicity. This can be explained by the increasing efficiency of grain growth in the interstellar medium (ISM) at higher metallicities, with a corollary being that the low DTM ratios seen at low metallicities are due to inefficient stellar dust production. This interpretation assumes that the efficiency of dust destruction in the ISM is constant, whereas it might be expected to increase at low metallicity; the decreased cooling efficiency of low-metallicity gas should result in more post-shock dust destruction via thermal sputtering. We show that incorporating a sufficiently strong metallicity dependence into models of galaxy evolution removes the need for low stellar dust yields. The contribution of stellar sources to the overall dust budget may be significantly underestimated, and that of grain growth overestimated, by models assuming a constant destruction efficiency.