We study a two dimensional dilaton gravity system, recently examined by Almheiri and Polchinski, which describes near extremal black holes, or more generally, nearly $AdS_2$ spacetimes. The asymptotic symmetries of $AdS_2$ are all the time reparametrizations of the boundary. These symmetries are spontaneously broken by the $AdS_2$ geometry and they are explicitly broken by the small deformation away from $AdS_2$. This pattern of spontaneous plus explicit symmetry breaking governs the gravitational backreaction of the system. It determines several gravitational properties such as the linear in temperature dependence of the near extremal entropy as well as the gravitational corrections to correlation functions. These corrections include the ones determining the growth of out of time order correlators that is indicative of chaos. These gravitational aspects can be described in terms of a Schwarzian derivative effective action for a reparametrization.