Hybrid plasmonic devices involve a nanostructured metal supporting localized surface plasmons to amplify light-matter interaction, and a non-plasmonic material to functionalize charge excitations. Application-relevant epitaxial heterostructures, however, give rise to ballistic ultrafast dynamics that challenge the conventional semiclassical understanding of unidirectional nanometal-to-substrate energy transfer. Epitaxial Au nanoislands are studied on WSe2 with time- and angle-resolved photoemission spectroscopy and femtosecond electron diffraction: this combination of techniques resolves material, energy, and momentum of charge-carriers and phonons excited in the heterostructure. A strong non-linear plasmon-exciton interaction that transfers the energy of sub-bandgap photons very efficiently to the semiconductor is observed, leaving the metal cold until non-radiative exciton recombination heats the nanoparticles on hundreds of femtoseconds timescales. The results resolve a multi-directional energy exchange on timescales shorter than the electronic thermalization of the nanometal. Electron-phonon coupling and diffusive charge-transfer determine the subsequent energy flow. This complex dynamics opens perspectives for optoelectronic and photocatalytic applications, while providing a constraining experimental testbed for state-of-the-art modelling.