Particle size measurement based on digital holography with conventional algorithms are usually time-consuming and susceptible to noises associated with hologram quality and particle complexity, limiting its usage in a broad range of engineering applications and fundamental research. We propose a learning-based hologram processing method to cope with the aforementioned issues. The proposed approach uses a modified U-net architecture with three input channels and two output channels, and specially-designed loss functions. The proposed method has been assessed using synthetic, manually-labeled experimental, and water tunnel bubbly flow data containing particles of different shapes. The results demonstrate that our approach can achieve better performance in comparison to the state-of-the-art non-machine-learning methods in terms of particle extraction rate and positioning accuracy with significantly improved processing speed. Our learning-based approach can be extended to other types of image-based particle size measurements.