The threshold displacement energy in solid state detector materials varies from several eV to ~100 eV. If a stable or long lived defect is created as a result of a nuclear recoil event, some part of the recoil energy is stored in the deformed lattice and is therefore not observable in a phonon detector. Thus, an accurate model of this effect is necessary for precise calibration of the recoil energy measurement in low threshold phonon detectors. Furthermore, the sharpness of the defect creation threshold varies between materials. For a hard material such as diamond, the sharp threshold will cause a sudden onset of the energy loss effect, resulting in a prominent peak in the observed recoil spectrum just below the threshold displacement energy. We describe how this effect can be used to discriminate between nuclear and electron recoils using just the measured recoil spectrum.