Visual inspection plays a crucial role during the manufacturing of tires which are essential for safe driving. Due to complicated anisotropic multi-texture background and ambiguous defect in it, automated tire defect detection is facing huge challenges and high costs. In this study, a novel two-stage convolutional neural network (CNN) is proposed for tire inspection by combining an optimized YOLOv3 and improved pyramid scene parsing network. Comparative experiments are conducted with the-state-of-the-art to validate the effectiveness and superior performance of the proposed method. The proposed two-stage CNN method achieves an average precision of 91.39%, the defect semantic segmentation achieves a mean intersection over union of 87.86%. The average detection time for a tire is 1.158 s such that the method can be effectively implemented into the industrial workflow. It can also be easily applied to different visual inspection applications.