Many large species have declined worldwide due to habitat fragmentation and poaching. The defaunation of large frugivores and the consequent reductions of seed dispersal services may have immediate effects on plant demography. Yet, the lasting effects of frugivore defaunation on microevolutionary processes of the plants they disperse remain understudied. We tested if the loss of large seed dispersers can lead to microevolutionary changes of a tropical palm. We show that frugivore defaunation is the main driver of changes in allelic frequency among populations. Turnover of alleles accounted for 100% of dissimilarity in allelic frequencies of individuals between defaunated and non-defaunated forests; and individuals from defaunated sites are 1.5 times more similar genetically than those found in pristine sites. Given that sizeable fractions of the palm fruit crops remain undispersed in defaunated sites due to lack of large-bodied frugivores, this distinct pattern of gene pool composition of early recruits may reveal strong dispersal limitation for specific genotypes, or collapses of gene flow between fragmented areas, or both. Because most of tropical tree species rely on seed dispersal by vertebrates, our results show that defaunation has a lasting effect on microevolutionary processes, with potential consequences for persistence under scenarios of environmental change.