Near term quantum hardware promises unprecedented computational advantage. Crucial in its development is the characterization and minimization of computational errors. We propose the use of the quantum fluctuation theorem to benchmark the accuracy of quantum annealers. This versatile tool provides simple means to determine whether the quantum dynamics are unital, unitary, and adiabatic, or whether the system is prone to thermal noise. Our proposal is experimentally tested on two generations of the D-Wave machine, which illustrates the sensitivity of the fluctuation theorem to the smallest aberrations from ideal annealing. In addition, for the optimally operating D-Wave machine, our experiment provides the first experimental verification of the integral fluctuation in an interacting, many-body quantum system.