Our goal is to understand primary accretion of the first planetesimals. Some examples are seen today in the asteroid belt, providing the parent bodies for the primitive meteorites. The primitive meteorite record suggests that sizeable planetesimals formed over a period longer than a million years, each of which being composed entirely of an unusual, but homogeneous, mixture of millimeter-size particles. We sketch a scenario that might help explain how this occurred, in which primary accretion of 10-100 km size planetesimals proceeds directly, if sporadically, from aerodynamically-sorted millimeter-size particles (generically "chondrules"). These planetesimal sizes are in general agreement with the currently observed asteroid mass peak near 100 km diameter, which has been identified as a "fossil" property of the pre-erosion, pre-depletion population. We extend our primary accretion theory to make predictions for outer Solar System planetesimals, which may also have a preferred size in the 100 km diameter range. We estimate formation rates of planetesimals and explore parameter space to assess the conditions needed to match estimates of both asteroid and Kuiper Belt Object (KBO) formation rates. For parameters that satisfy observed mass accretion rates of Myr-old protoplanetary nebulae, the scenario is roughly consistent with not only the "fossil" sizes of the asteroids, and their estimated production rates, but also with the observed spread in formation ages of chondrules in a given chondrite, and with a tolerably small radial diffusive mixing during this time between formation and accretion. As previously noted, the model naturally helps explain the peculiar size distribution of chondrules within such objects. The optimum range of parameters, however, represents a higher gas density and fractional abundance of solids, and a smaller difference between Keplerian and pressure-supported orbital velocities, than "canonical" models of the solar nebula. We discuss several potential explanations for these differences. The scenario also produces 10-100 km diameter primary KBOs, and also requires an enhanced abundance of solids to match the mass production rate estimates for KBOs (and presumably the planetesimal precursors of the ice giants themselves). We discuss the advantages and plausibility of the scenario, outstanding issues, and future directions of research.