We propose a crack detection network based on an image segmentation network for robust crack detection, which utilizes information from the entire image and performs pixel-wise prediction. To overcome the lack of data, we also propose a crack image generation algorithm using a 2D Gaussian kernel and the Brownian motion process. We gathered 242 crack images from plain images to cluttered images to train and verify the robustness of the proposed crack segmentation network. To verify the usefulness of simulated cracks, we used 2 integrated datasets constructed with 100 and 200 simulated crack images added to the actual crack dataset, as well as an actual crack dataset. To derive the maximum prediction performance, the neural network was pre-trained on the MS-COCO dataset, and re-trained by each crack dataset. The results show that the proposed method is highly robust and accurate, even for complex images. The trained network was also tested under different brightness, hue, and noise conditions, and results have shown that this promising method can be used in various inspection platforms.