Spectroscopy is a unique experimental tool for measuring the fundamental Casimir-Polder interaction between excited state atoms, or other polarizable quantum objects, and a macroscopic surface. Spectroscopic measurements probe atoms at nanometric distances away from the surface where QED retardation is usually negligible and the atom-surface interaction is proportional to the inverse cube of the separation distance, otherwise known as the van der Waals regime. Here we focus on selective reflection, one of the main spectroscopic probes of Casimir-Polder interactions. We calculate for the first time selective reflection spectra using the full, distance dependent, Casimir-Polder energy shift and linewidth. We demonstrate that retardation can have significant effects, in particular for experiments with low-lying energy states. We also show that the effective probing depth of selective reflection spectroscopy depends on the transition linewidth. Our analysis allows us to calculate selective reflection spectra with composite surfaces, such as metasurfaces, dielectric stacks, or even bi-dimensional materials.