The role of medium-induced relaxation of intermediate (bridge) sites in energy and charge transfer processes in molecular aggregates of arbitrary size and geometry is explored by means of Green's function techniques. The coupling of electronic and (solvent and intramolecular) nuclear degrees of freedom is incorporated using the Brownian oscillator model, which allows an exact calculation of the necessary two-point and four-point correlation functions of exciton operators. The signatures of energy transfer and spectral diffusion in time- and frequency-resolved fluorescence spectroscopy are studied. A unified expression for the frequency-dependent transfer rate is derived, which interpolates between the sequential and superexchange limits. Numerical results and a Liouville space pathway analysis for a donor-acceptor system coupled through a single bridge molecule are presented.