Dark matter (DM) constitutes 85% of the matter in the Universe. However, its specific particle property is still unclear. The fundamentals of DM particles subject to gravitational interaction, and that the lepton excess in cosmic rays may originate from DM particles, inspired us to investigate DM particle properties beyond the standard model. We assume that a leptophilic SU(2) doublet exists in nature as the mediator connecting DM with visible leptons. Since general relativity is not renormalizable at the quantum level, it should be regarded as an effective field theory's leading order term. One species of the next-to-leading-order term should be operators linear to the Ricci scalar and containing scalar fields, such as the Higgs field, scalar DM, or the newly introduced SU(2) scalar doublet. These operators can cause DM annihilation through gravity portals. We analyzed constraints from the cosmic antiproton flux, DM relic abundance, cosmic positron flux, cosmic microwave background, and direct detection experiments. The result shows that there is a vast parameter space that is compatible with current experiments. DM with a mass of electroweak scale is only allowed to annihilate into leptons. We further show that the purely gravitational DM better explains the DArk Matter Particle Explorer cosmic lepton excess. Our work provides a promising mechanism for DM particles to connect with standard model particles.