Femtosecond time-resolved two-photon photoemission (2PPE) spectroscopy is employed to investigate the electron dynamics of image potential states on physisorbed Xe and N2 overlayers adsorbed on Cu(111). With increasing number of adsorbate layers the binding energies of the image potential states are found to decrease while their lifetimes increase (except for the second image potential state on 2 to 3 ML Xe/Cu(111)). These trends are most pronounced for nitrogen, where the binding energy of the first image potential state decreases by a factor of 3.5 from 0 to 2 ML N2/Cu(111); at the same time the lifetime increases from 22 to 700 fs. The results are discussed in the framework of the dielectric continuum model, which approximates the adsorbate layers by a dielectric slab in front of the metal surface. For Xe, the agreement between measured and calculated lifetimes improves significantly if the full dispersion curve of the Xe6s conduction band is taken into account.