The first visible-light studies of Earth-sized extrasolar planets will employ photometry or low-resolution spectroscopy. This work uses EPOCh medium-band filter photometry between 350 and 950 nm obtained with the Deep Impact (DI) High Resolution Instrument (HRI) of Earth, the Moon, and Mars in addition to previous full-disk observations of the other six solar system planets and Titan to analyze the limitations of using photometric colors to characterize extrasolar planets. We determined that the HRI 350, 550, and 850 nm filters are optimal for distinguishing Earth from the other planets and separating planets to first order based on their atmospheric and surface properties. Detailed conclusions that can be drawn about exoplanet atmospheres simply from a color-color plot are limited due to potentially competing physical processes in the atmosphere. The presence of a Rayleigh scattering atmosphere can be detected by an increase in the 350-550 nm brightness ratio, but the absence of Rayleigh scattering cannot be confirmed due to the existence of atmospheric and surface absorbing species in the UV. Methane and ammonia are the only species responsible for strong absorption in the 850 nm filter in our solar system. The combination of physical processes present on extrasolar planets may differ from those we see locally. Nevertheless, a generation of telescopes capable of collecting such photometric observations can serve a critical role in first-order characterization and constraining the population of Earth-like extrasolar planets.