Technologies that rely on quantum bits (qubits) require long coherence times and high-fidelity operations1. Superconducting qubits are one of the leading platforms for achieving these objectives2,3. However, the coherence of superconducting qubits is affected by the breaking of Cooper pairs of electrons4-6. The experimentally observed density of the broken Cooper pairs, referred to as quasiparticles, is orders of magnitude higher than the value predicted at equilibrium by the Bardeen-Cooper-Schrieffer theory of superconductivity7-9. Previous work10-12 has shown that infrared photons considerably increase the quasiparticle density, yet even in the best-isolated systems, it remains much higher10 than expected, suggesting that another generation mechanism exists13. Here we provide evidence that ionizing radiation from environmental radioactive materials and cosmic rays contributes to this observed difference. The effect of ionizing radiation leads to an elevated quasiparticle density, which we predict would ultimately limit the coherence times of superconducting qubits of the type measured here to milliseconds. We further demonstrate that radiation shielding reduces the flux of ionizing radiation and thereby increases the energy-relaxation time. Albeit a small effect for today's qubits, reducing or mitigating the impact of ionizing radiation will be critical for realizing fault-tolerant superconducting quantum computers.