We include the relativistic lattice KP hierarchy, introduced by Gibbons and Kupershmidt, into the $r$-matrix framework. An $r$-matrix account of the nonrelativistic lattice KP hierarchy is also provided for the reader's convenience. All relativistic constructions are regular one-parameter perturbations of the nonrelativistic ones. We derive in a simple way the linear Hamiltonian structure of the relativistic lattice KP, and find for the first time its quadratic Hamiltonian structure. Amasingly, the latter turns out to coincide with its nonrelativistic counterpart (a phenomenon, known previously only for the simplest case of the relativistic Toda lattice).