Macrophages internalize pathogens for intracellular degradation. An important part of this process is the phagosomal transport from the cell periphery to the perinuclear region. Biochemical factors are known to influence the fate of phagosomes. Here, we show that the size of phagosomes also has a strong influence on their transport. We found that large phagosomes are transported persistently to the nucleus, whereas small phagosomes show strong bidirectional transport. We show that dynein motors play a larger role in the transport of large phagosomes, whereas actin filament-based motility plays a larger role in the transport of small phagosomes. Furthermore, we investigated the spatial distribution of dyneins and microtubules around phagosomes and hypothesize that dynein and microtubule density differences between the nucleus-facing side of phagosomes and the opposite side could explain part of the observed transport characteristics. Our findings suggest that a size-dependent cellular sorting mechanism might exist that supports macrophages in their immunological roles.