We report time domain observation of vacuum Rabi oscillations in a single quantum dot strongly coupled to a nanocavity under incoherent optical carrier injection. We realize a photonic crystal nanocavity with a very high quality factor of >80 000 and employ it to clearly resolve the ultrafast vacuum Rabi oscillations by simple photoluminescence-based experiments. We found that the time domain vacuum Rabi oscillations were largely modified when changing the pump wavelength and intensity, even when marginal changes were detected in the corresponding photoluminescence spectra. We analyze the measured time domain oscillations by fitting to simulation curves obtained with a cavity quantum electrodynamics model. The observed modifications of the oscillation curves were mainly induced by the change in the carrier capture and dephasing dynamics in the quantum dot, as well as the change in bare-cavity emission. This result suggests that vacuum Rabi oscillations can be utilized as a highly sensitive probe for the quantum-dot dynamics. Our work points out a powerful alternative to conventional spectral-domain measurements for a deeper understanding of the vacuum Rabi dynamics in quantum-dot-based-cavity quantum electrodynamics systems.