Neurons in macaque ventral premotor cortex and inferior parietal lobe discharge during both the observation and the execution of motor acts. It has been claimed that these so-called mirror neurons form the basis of action understanding by matching the visual input with the corresponding motor program (direct matching). Functional magnetic resonance imaging (fMRI) adaptation can be used to test the direct matching account of action recognition by determining whether putative mirror neurons show adaptation for repeated motor acts independently of whether they are observed or executed. An unambiguous test of the hypothesis requires that the motor acts be meaningless to ensure that any adaptation effect is directly because of movement recognition/motor execution and not contextually determined inferences. We found adaptation for motor acts that were repeatedly observed or repeatedly executed. We also found adaptation for motor acts that were first observed and then executed, as would be expected if a previously seen act primed the subsequent execution of that act. Crucially, we found no signs of adaptation for motor acts that were first executed and then observed. Failure to find cross-modal adaptation for executed and observed motor acts is not compatible with the core assumption of mirror neuron theory, which holds that action recognition and understanding are based on motor simulation.