Numerical solutions to the Vlasov-Poisson system of equations have important applications to both plasma physics and cosmology. In this paper, we present a new Particle-in-Cell (PIC) method for solving this system that is 4th-order accurate in both space and time. Our method is a high-order extension of one presented previously [B. Wang, G. Miller, and P. Colella, SIAM J. Sci. Comput., 33 (2011), pp. 3509--3537]. It treats all of the stages of the standard PIC update - charge deposition, force interpolation, the field solve, and the particle push - with 4th-order accuracy, and includes a 6th-order accurate phase-space remapping step for controlling particle noise. We demonstrate the convergence of our method on a series of one- and two- dimensional electrostatic plasma test problems, comparing its accuracy to that of a 2nd-order method. As expected, the 4th-order method can achieve comparable accuracy to the 2nd-order method with many fewer resolution elements.