ESO 4 m class telescope and VLT deep imaging of the isolated neutron star RX J0720.4-3125 reveals a proper motion of mu = 97 ± 12 mas/yr and a blue U-B color index. We show that a neutron star atmosphere model modified to account for a limited amount of hydrogen on the star's surface can well represent both the optical and X-ray data without invoking any additional thermal component. The large proper motion almost completely excludes the possibility that accretion from the interstellar medium is the powering mechanism of the X-ray emission. It also implies that the proposed spin down is entirely due to magnetic dipole losses. RX J0720.4-3125 is thus very likely a middle aged cooling neutron star. Its overall properties are quite similar to some of the long period radio pulsars recently discovered, giving further support to the idea that RX J0720.4-3125 may be a pulsar whose narrow radio beam does not cross the Earth.