Continents ride high above the ocean floor because they are underlain by thick, low-density, Si-rich, and Mg-poor crust. However, the parental magmas of continents were basaltic, which means they must have lost Mg relative to Si during their maturation into continents. Igneous differentiation followed by lower crustal delamination and chemical weathering followed by subduction recycling are possible solutions, but the relative magnitudes of each process have never been quantitatively constrained because of the lack of appropriate data. Here, we show that the relative contributions of these processes can be obtained by simultaneous examination of Mg and Li (an analog for Mg) on the regional and global scales in arcs, delaminated lower crust, and river waters. At least 20% of Mg is lost from continents by weathering, which translates into >20% of continental mass lost by weathering (40% by delamination). Chemical weathering leaves behind a more Si-rich and Mg-poor crust, which is less dense and hence decreases the probability of crustal recycling by subduction. Net continental growth is thus modulated by chemical weathering and likely influenced by secular changes in weathering mechanisms.