We analyze the B-RC colors of galaxies as functions of luminosity and local galaxy density using a large photometric redshift catalog based on the Red-Sequence Cluster Survey. We select two samples of galaxies with a magnitude limit of MRC<-18.5 and redshift ranges of 0.2<=z<0.4 and 0.4<=z<0.6 containing ~105 galaxies each. We model the color distributions of subsamples of galaxies and derive the red galaxy fraction and peak colors of red and blue galaxies as functions of galaxy luminosity and environment. The evolution of these relationships over the redshift range of z~0.5-0.05 is analyzed in combination with published results from the Sloan Digital Sky Survey. We find that there is a strong evolution in the rest-frame peak color of bright blue galaxies in that they become redder with decreasing redshift, while the colors of faint blue galaxies remain approximately constant. This effect supports the ``downsizing'' scenario of star formation in galaxies. While the general dependence of the galaxy color distributions on the environment is small, we find that the change of red galaxy fraction with epoch is a function of the local galaxy density, suggesting that the downsizing effect may operate with different timescales in regions of different galaxy densities.Based on observations from the Canada-France-Hawaii Telescope, which is operated by the National Research Council of Canada, the Centre Nationale de la Recherche Scientifique, and the University of Hawaii.