The kinetic Sunyaev Zel'dovich (kSZ) and moving lens effects, secondary contributions to the cosmic microwave background (CMB), carry significant cosmological information due to their dependence on the large-scale peculiar velocity field. Previous work identified a promising means of extracting this cosmological information using a set of quadratic estimators for the radial and transverse components of the velocity field. These estimators are based on the statistically anisotropic components of the cross-correlation between the CMB and a tracer of large scale structure, such as a galaxy redshift survey. In this work, we assess the challenges to the program of velocity reconstruction posed by various foregrounds and systematics in the CMB and galaxy surveys, as well as biases in the quadratic estimators. To do so, we further develop the quadratic estimator formalism and implement a numerical code for computing properly correlated spectra for all the components of the CMB (primary/secondary blackbody components and foregrounds) and a photometric redshift survey, with associated redshift errors, to allow for accurate forecasting. We create a simulation framework for generating realizations of properly correlated CMB maps and redshift binned galaxy number counts, assuming the underlying fields are Gaussian, and use this to validate a velocity reconstruction pipeline and assess map-based systematics such as masking. We highlight the most significant challenges for velocity reconstruction, which include biases associated with: modelling errors, characterization of redshift errors, and coarse graining of cosmological fields on our past light cone. Despite these challenges, the outlook for velocity reconstruction is quite optimistic, and we use our reconstruction pipeline to confirm that these techniques will be feasible with near-term CMB experiments and photometric galaxy redshift surveys.