Migrations have played an important role in shaping the genetic diversity of human populations. Understanding genomic data thus requires careful modeling of historical gene flow. Here we consider the effect of relatively recent population structure and gene flow, and interpret genomes of individuals that have ancestry from multiple source populations as mosaics of segments originating from each population. We propose general and tractable models for describing the evolution of these patterns of local ancestry and their impact on genetic diversity. We focus on the length distribution of continuous ancestry tracts, and the variance in total ancestry proportions among individuals. The proposed models offer improved agreement with Wright-Fisher simulation data when compared to state-of-the art models, and can be used to infer various demographic parameters in gene flow models. Considering HapMap African-American (ASW) data, we find that a model with two distinct phases of `European' gene flow significantly improves the modeling of both tract lengths and ancestry variances.