Here we present modeling of the polarization aberrations of the Gemini telescope as seen by the Gemini Planet Imager's dual-channel polarimetry mode. We first calculate the Jones pupil using a polarization sensitive ray-tracing program and theoretical properties for the Gemini mirror coatings. The predominant effect is a differential astigmatism between incident x and y polarization states, also known as retardance defocus. We use this Jones pupil as an input to a Fourier optics model of the Gemini Planet Imager to create model Stokes Q and U images, which we compare to on-sky data. We demonstrate that the effects of the telescope polarization aberrations can be detected in the on-sky data and speculate that they may be limiting the polarimetric contrasts at the smallest inner working angles. Finally, we explore how the polarization aberrations couple with residual atmospheric wavefront error from the adaptive optics system, and how they affect the polarimetric performance of new GPI 2.0 coronagraph designs.