Planet formation is sensitive to the conditions in protoplanetary disks, for which scaling laws as a function of stellar mass are known. We aim to test whether the observed population of planets around low-mass stars can be explained by these trends, or if separate formation channels are needed. We address this question by confronting a state-of-the-art planet population synthesis model with a sample of planets around M dwarfs observed by the HARPS and CARMENES radial velocity (RV) surveys. To account for detection biases, we performed injection and retrieval experiments on the actual RV data to produce synthetic observations of planets that we simulated following the core accretion paradigm. These simulations robustly yield the previously reported high occurrence of rocky planets around M dwarfs and generally agree with their planetary mass function. In contrast, our simulations cannot reproduce a population of giant planets around stars less massive than 0.5 solar masses. This potentially indicates an alternative formation channel for giant planets around the least massive stars that cannot be explained with current core accretion theories. We further find a stellar mass dependency in the detection rate of short-period planets. A lack of close-in planets around the earlier-type stars ($M_\star \gtrsim 0.4\, M_\odot$) in our sample remains unexplained by our model and indicates dissimilar planet migration barriers in disks of different spectral subtypes. Both discrepancies can be attributed to gaps in our understanding of planet migration in nascent M dwarf systems. They underline the different conditions around young stars of different spectral subtypes, and the importance of taking these differences into account when studying planet formation.