We propose a novel source of gravitational wave emission: the inspirals of compact fragments inside primordial supermassive stars (SMSs). Such systems are thought to be an essential channel in the as-yet little understood formation of supermassive black holes (SMBHs). One model suggests that high accretion rates of $0.1$-1 M$_\odot$/yr attainable in atomically-cooled primordial halos can lead to the formation of a nuclear-burning SMS. This will ultimately undergo collapse through a relativistic instability, leaving a massive BH remnant. Recent simulations suggest that supermassive stars rarely form in isolation, and that companion stars and even black holes formed may be captured/accreted and inspiral to the SMS core due to gas dynamical friction. Here, we explore the GW emission produced from such inspirals, which could probe the formation and evolution of SMS and seeds of the first supermassive black holes. We use a semi-analytic gas-dynamical friction model of the inspirals in the SMS to characterize their properties. We find such sources could potentially be observable by upcoming space-born GW-detectors at their formation redshifts with the benefit of gravitational lensing. Mergers within closely-related quasi-stars may produce a much stronger signal, though disambiguating such events from other high-z events may prove challenging.