Radiation recoil (YORP) torques are shown to be extremely sensitive to small-scale surface topography, using numerical simulations. Starting from a set of "base objects" representative of the near-Earth object population, random realizations of three types of small-scale topography are added: Gaussian surface fluctuations, craters, and boulders. For each, the expected relative errors in the spin and obliquity components of the YORP torque caused by the observationally unresolved small-scale topography are computed. Gaussian power, at angular scales below an observational limit, produces expected errors of order 100% if observations constrain the surface to a spherical harmonic order l≲10. For errors under 10%, the surface must be constrained to at least l=20. A single crater with diameter roughly half the object's mean radius, placed at random locations, results in expected errors of several tens of percent. The errors scale with crater diameter D as D for D>0.3 and as D for D<0.3 mean radii. Objects that are identical except for the location of a single large crater can differ by factors of several in YORP torque, while being photometrically indistinguishable at the level of hundredths of a magnitude. Boulders placed randomly on identical base objects create torque errors roughly 3 times larger than do craters of the same diameter, with errors scaling as the square of the boulder diameter. A single boulder comparable to Yoshinodai on 25143 Itokawa, moved by as little as twice its own diameter, can alter the magnitude of the torque by factors of several, and change the sign of its spin component at all obliquities. Most of the total torque error produced by multiple unresolved craters is contributed by the handful of largest craters; but both large and small boulders contribute comparably to the total boulder-induced error. A YORP torque prediction derived from groundbased data can be expected to be in error by of order 100% due to unresolved topography. Small surface changes caused by slow spin-up or spin-down may have significant stochastic effects on the spin evolution of small bodies. For rotation periods between roughly 2 and 10 h, these unpredictable changes may reverse the sign of the YORP torque. Objects in this spin regime may random-walk up and down in spin rate before the rubble-pile limit is exceeded and fissioning or loss of surface objects occurs. Similar behavior may be expected at rotation rates approaching the limiting values for tensile-strength dominated objects.