A time-domain spectroscopic technique, based on the generation and detection of a collimated beam of subpicosecond broadband terahertz pulses, is used to measure the absorption and dispersion of n- and p-type silicon, with resistivities of 0.1, 1, and 10 Ω cm in the submillimeter range of 0.1-2 THz. From the transmission measurements performed at room temperature and at 80 K, the absorption and dispersion, and concomitantly the full complex conductivity, of the doped silicon could be obtained. The results provide an accurate view on the dynamics of the electrons and the holes. Although the simple Drude model, with an energy-independent relaxation time, gives a surprisingly accurate description of the observed carrier dynamics, the measurements do show that some refinements are needed. An extended model, with an energy-dependent carrier-relaxation rate, can explain most of the observed deviations from the simple Drude model.