A single-channel color image encryption is proposed based on a phase retrieve algorithm and a two-coupled logistic map. Firstly, a gray scale image is constituted with three channels of the color image, and then permuted by a sequence of chaotic pairs generated by the two-coupled logistic map. Secondly, the permutation image is decomposed into three new components, where each component is encoded into a phase-only function in the fractional Fourier domain with a phase retrieve algorithm that is proposed based on the iterative fractional Fourier transform. Finally, an interim image is formed by the combination of these phase-only functions and encrypted into the final gray scale ciphertext with stationary white noise distribution by using chaotic diffusion, which has camouflage property to some extent. In the process of encryption and decryption, chaotic permutation and diffusion makes the resultant image nonlinear and disorder both in spatial domain and frequency domain, and the proposed phase iterative algorithm has faster convergent speed. Additionally, the encryption scheme enlarges the key space of the cryptosystem. Simulation results and security analysis verify the feasibility and effectiveness of this method.