Early-type stars show a bimodal distribution of magnetic field strengths, with some showing very strong fields (≳1 kG) and others very weak fields (≲10 G). Recently, we proposed that this reflects the processing or lack thereof of fossil fields by subsurface convection zones. Stars with weak fossil fields process these at the surface into even weaker dynamo-generated fields, while in stars with stronger fossil fields magnetism inhibits convection, allowing the fossil field to remain as is. We now expand on this theory and explore the timescales involved in the evolution of near-surface magnetic fields. We find that mass loss strips near-surface regions faster than magnetic fields can diffuse through them. As a result, observations of surface magnetism directly probe the frozen-in remains of the convective dynamo. This explains the slow evolution of magnetism in stars with very weak fields: these dynamo-generated magnetic fields evolve on the timescale of the mass loss, not that of the dynamo.