Light field microscopy (LFM) uses a microlens array (MLA) near the sensor plane of a microscope to achieve single-shot 3D imaging of a sample, without any moving parts. Unfortunately, the 3D capability of LFM comes with a significant loss of lateral resolution at the focal plane, which is highly undesirable in microscopy. Placing the MLA near the pupil plane of the microscope, rather than the image plane, can mitigate the artifacts near focus and provide an efficient shift-invariant model at the expense of field-of-view. Here, we show that our Fourier DiffuserScope achieves significantly better performance than Fourier LFM. Fourier DiffuserScope uses a diffuser in the pupil plane to encode depth information, then reconstructs volumetric information computationally by solving a sparsity-constrained inverse problem. Our diffuser consists of randomly placed microlenses with varying focal lengths. We show that by randomizing the microlens positions, a larger lateral field-of-view can be achieved compared to a conventional MLA; furthermore, by adding diversity to the focal lengths, the axial depth range is increased. To predict system performance based on diffuser parameters, we for the first time establish a theoretical framework as a design guideline, followed by numerical simulations to verify. Under both theoretical and numerical analysis, we demonstrate that our diffuser design provides more uniform resolution over a larger volume, both laterally and axially, outperforming the MLA used in LFM. We build an experiment system with an optimized lens set and achieve < 3 um lateral and 4 um axial resolution over a 1000 x 1000 x 280 um^3 volume.