A small fraction of early-type galaxies (ETGs) show prolate rotation, i.e. they rotate around their long photometric axis. In simulations, certain configurations of galaxy mergers are known to produce this type of rotation. We investigate the association of prolate rotation and signs of galaxy interactions among the observed galaxies. We collected a sample of 19 nearby ETGs with distinct prolate rotation from the literature and inspected their ground-based deep optical images for interaction signs - 18 in archival images and one in a new image obtained with the Milanković telescope. Tidal tails, shells, asymmetric/disturbed stellar halos, or on-going interactions are present in all the 19 prolate rotators. Comparing this with the frequency of tidal disturbance among the general sample of ETGs of a roughly similar mass range and surface-brightness limit, we estimate that the chance probability of such an observation is only 0.00087. We also found a significant overabundance of prolate rotators that are hosting multiple stellar shells. The visible tidal features imply a relatively recent galaxy interaction. That agrees with the Illustris large-scale cosmological hydrodynamical simulation, where prolate rotators are predominantly formed in major mergers during the last 6 Gyr. In the appendix, we present the properties of an additional galaxy, NGC 7052, a prolate rotator for which no deep images are available, but for which an HST image revealed the presence of a prominent shell, which had not been reported before.