We present a scheme for deterministic ion-photon qubit exchange, namely a SWAP gate, based on realistic cavity-QED systems with 171Yb+, 40Ca+ and 138Ba+ ions. The gate can also serve as a single-photon quantum memory, in which an outgoing photon heralds the successful arrival of the incoming photonic qubit. Although strong coupling, namely having the single-photon Rabi frequency be the fastest rate in the system, is often assumed essential, this gate (similarly to the Duan-Kimble C-phase gate) requires only Purcell enhancement, i.e. high single-atom cooperativity. Accordingly, it does not require small mode volume cavities, which are challenging to incorporate with ions due to the difficulty of trapping them close to dielectric surfaces. Instead, larger cavities, potentially more compatible with the trap apparatus, are sufficient, as long as their numerical aperture is high enough to maintain small mode area at the ion's position. We define the optimal parameters for the gate's operation and simulate the expected fidelities and efficiencies, demonstrating that efficient photon-ion qubit exchange, a valuable building block for scalable quantum computation, is practically attainable with current experimental capabilities.