We study the relation between the UV-slope, $\beta$, and the ratio between the IR and UV luminosities of galaxies, IRX, using TNG50, the latest installment of the IllustrisTNG galaxy formation simulations. We select 7280 TNG50 star-forming galaxies with stellar mass above $10^9M_\odot$ at selected redshifts, $0 \leq z \leq 4$, and perform radiative transfer calculations with SKIRT to model the effects of ISM dust on the simulated stellar light. We adopt a Milky Way (MW) type dust and a fixed dust-to-metal ratio of 0.3 throughout and find that TNG50 star-forming main-sequence galaxies agree with the empirically-derived reference IRX-$\beta$ relations at $z \lesssim 1$. There appears to be a redshift-dependent systematic offset with respect to the reference relations, with the TNG50 IRX-$\beta$ median relation steepening towards higher redshifts. This is to first order driven by variations in intrinsic UV-slopes due to different star-formation histories of galaxies selected at different cosmic epochs. Once differences in the intrinsic UV-slope are corrected for, TNG50 galaxies exhibit a scatter of 0.3 dex in IRX at fixed $\beta$ at all studied redshifts. These galaxy-to-galaxy variations are correlated with intrinsic galaxy properties, such that at fixed UV-slope galaxies with higher star--formation rates, star--formation efficiencies, gas metallicities, and stellar masses exhibit larger IRX values. We additionally demonstrate a degeneracy between stellar age and dust type. The combination of young stellar population age with a SMC type of dust for high-redshift ($z=4$) TNG50 galaxies can result in an IRX-$\beta$ relation similar to what is found for low-redshift galaxies with MW dust. This hampers the use of the IRX-$\beta$ relation as a proxy for dust type. We provide a redshift-dependent fitting function for the IRX-$\beta$ relation with MW dust based on our numerical models.