Dwarf galaxies (M*<10^9 Msun) are key drivers of mass assembly in high mass galaxies, but relatively little is understood about the assembly of dwarf galaxies themselves. Using the New Horizon cosmological simulation (40 pc spatial resolution), we investigate how mergers and fly-bys drive the mass assembly and structural evolution of around 1000 field and group dwarfs up to z=0.5. We find that, while dwarf galaxies often exhibit disturbed morphologies (30 and 50 per cent are disturbed at z=1 and z=3 respectively), only a small proportion of the morphological disturbances seen in dwarf galaxies are driven by mergers at any redshift (for 10^9 Msun, mergers drive only 20 per cent morphological disturbances). They are instead primarily the result of interactions that do not end in a merger (e.g. fly-bys). Given the large fraction of apparently morphologically disturbed dwarf galaxies which are not, in fact, merging, this finding is particularly important to future studies identifying dwarf mergers and post-mergers morphologically at intermediate and high redshifts. Dwarfs typically undergo one major and one minor merger between z=5 and z=0.5, accounting for 10 per cent of their total stellar mass. Mergers can also drive moderate star formation enhancements at lower redshifts (up to 5 times at z=1), but this only accounts for 5 per cent of stellar mass in the dwarf regime given their infrequency. Non-merger interactions drive significantly smaller star formation enhancements (around two times), but their preponderance relative to mergers means they account for 10 per cent of stellar mass in the dwarf regime.