Based on an effective Lagrangian which combines chiral SU(3) dynamics with vector meson dominance, we have developed a model for the forward vector meson-nucleon scattering amplitudes. We use this as an input to calculate the low energy part of the current-current correlation function in nuclear matter. Its spectrum enters directly in the "left-hand side" of QCD sum rules. For the isovector channel we find a significant enhancement of the in-medium spectral density below the ϱ resonane while the ρ meson mass itself changes only slightly. The situation is different in the isoscalar channel, where the mass and peak position of the ω meson move downward while its width increases less drastically than in the ρ meson case. For the φ meson we find almost no mass shift; the width of the peak broadens moderately. We observe a remarkable degree of consistency with the operator product expansion of QCD sum rules in all three channels. We point out, however, that these results cannot simply be interpreted, as commonly done, in terms of a universal rescaling of vector meson masses in matter.