It is now widely accepted that many X-ray binaries (XRBs) form in compact star clusters. Therefore, it follows that there may be a link between the properties of star clusters and the production of XRBs. We study the masses and densities of compact clusters with different ages found in the merging Antennae galaxies and in the dwarf starburst NGC 4449, to assess their impact on the production of XRBs. The very youngest clusters, with ages log (τ/yr) < 7.0, likely host high-mass XRBs, while older clusters with ages log (τ/yr) = 7.0-8.6 host intermediate-mass XRBs. We find tentative evidence that intermediate-mass XRBs are more likely to form in more massive and dense clusters, while high-mass XRBs do not appear to form in clusters that favor higher masses or densities. If confirmed, these findings support a picture where high-mass and intermediate-mass XRBs form through different mechanisms.