Whispering-gallery mode (WGM) resonators are an important platform for linear, nonlinear, and quantum optical experiments. In such experiments, independent control of in-coupling and out-coupling rates to different modes can lead to higher conversion efficiencies and greater flexibility in the generation of nonclassical states based on parametric down-conversion. In this work, we introduce a scheme that enables selective out-coupling of WGMs belonging to a specific polarization family, while the orthogonally polarized modes remain largely unperturbed. Our technique utilizes material birefringence in both the resonator and the coupler such that a negative (positive) birefringence allows for polarization-selective coupling to TE (TM) WGMs. We formulate a refined coupling condition suitable for describing the case where the refractive indices of the resonator and the coupler are almost the same, from which we derive a criterion for polarization-selective coupling. Finally, we experimentally demonstrate our proposed method using a lithium niobate disk resonator coupled to a lithium niobate prism, where we show a 22-dB suppression of coupling to TM modes relative to TE modes.