Domestication is a multifaceted evolutionary process, involving changes in individual genes, genetic interactions, and emergent phenotypes. There has been extensive discussion of the phenotypic characteristics of plant domestication, and recent research has started to identify the specific genes and mutational mechanisms that control domestication traits. However, there is an apparent disconnect between the simple genetic architecture described for many crop domestication traits, which should facilitate rapid phenotypic change under selection, and the slow rate of change reported from the archeobotanical record. A possible explanation involves the middle ground between individual genetic changes and their expression during development, where gene-by-gene (epistatic) and gene-by-environment interactions can modify the expression of phenotypes and opportunities for selection. These aspects of genetic architecture have the potential to significantly slow the speed of phenotypic evolution during crop domestication and improvement. Here we examine whether epistatic and gene-by-environment interactions have shaped how domestication traits have evolved. We review available evidence from the literature, and we analyze two domestication-related traits, shattering and flowering time, in a mapping population derived from a cross between domesticated foxtail millet and its wild progenitor. We find that compared with wild progenitor alleles, those favored during domestication often have large phenotypic effects and are relatively insensitive to genetic background and environmental effects. Consistent selection should thus be able to rapidly change traits during domestication. We conclude that if phenotypic evolution was slow during crop domestication, this is more likely due to cultural or historical factors than epistatic or environmental constraints.