The flux anomalies in four-image gravitational lenses can be interpreted as evidence for the dark matter substructure predicted by cold dark matter (CDM) halo models. In principle, these flux anomalies could arise from alternate sources, such as absorption, scattering, or scintillation by the interstellar medium (ISM) of the lens galaxy, problems in the ellipsoidal macro models used to fit lens systems, or stellar microlensing. We apply several tests to the data that appear to rule out these alternate explanations. First, the radio flux anomalies show no significant dependence on wavelength, as would be expected for almost any propagation effect in the ISM or microlensing by the stars. Second, the flux anomaly distributions show the characteristic demagnifications of the brightest saddle point relative to the other images expected for low optical depth substructure, which cannot be mimicked by either the ISM or problems in the macro models. Microlensing by stars also cannot reproduce the suppression of the bright saddle points if the radio source sizes are consistent with the Compton limit for their angular sizes. Third, while it is possible to change the smooth lens models to fit the flux anomalies in some systems, we can rule out the necessary changes in all systems where we have additional lens constraints to check the models. Moreover, the parameters of these models are inconsistent with our present observations and expectations for the structure of galaxies. We conclude that low-mass halos remain the best explanation of the phenomenon.