HST images of the young supernova remnant Cas A are used to explore the expansion and spatial distribution of its highest velocity debris. ACS WFC images taken in 2004 March and December with Sloan F625W, F775W, and F850LP filters were used to identify 1825 high-velocity, outlying ejecta knots through measured proper motions of 0.35"-0.90" yr-1, corresponding to Vtrans=5500-14,500 km s-1 assuming d=3.4 kpc. The distribution of derived transverse expansion velocities for these ejecta knots shows a striking bipolar asymmetry with the highest velocity knots (Vtrans>=10,500 km s-1) confined to nearly opposing northeast and southwest ``jets'' at P.A.=45deg-70deg and 230°-270°, respectively. The jets have about the same maximum expansion velocity of ~=14,000 km s-1 and appear kinematically and chemically distinct in that they are the remnant's only S-rich ejecta with expansion velocities above the 10,000-11,000 km s-1 exhibited by outer nitrogen-rich ejecta, which otherwise represent the remnant's highest velocity debris. In addition, we find significant gaps in the spatial distribution of outlying ejecta in directions that are approximately perpendicular to the jets (P.A.=145deg-200deg and 335°-350°). The remnant's central X-ray point source lies some 7" to the southeast of the estimated expansion center (P.A.=169deg+/-8.4d) indicating a projected motion toward the middle of the broad southern ejecta knot gap. Extrapolations of measured 9 month proper motions for all 1825 outer ejecta knots and a selected subsample of 72 bright and compact knots suggest explosion dates (assuming no knot deceleration) of 1662+/-27 and 1672+/-18, respectively. We find some evidence for nonuniform deceleration in different directions around the remnant and find 126 knots located along the northwestern limb among the least decelerated ejecta, suggesting a convergence date of 1681+/-19. A remnant age of around 325 yr would imply a ~=350 km s-1 transverse velocity for the central X-ray point source.Based on observations with the NASA/ESA Hubble Space Telescope, obtained at the Space Telescope Science Institute, which is operated by the Association of Universities for Research in Astronomy (AURA), Inc., under NASA contract NAS5-26555.