A group $\Gamma$ is said to be uniformly HS stable if any map $\varphi : \Gamma \to U(n)$ that is almost a unitary representation (w.r.t. the Hilbert Schmidt norm) is close to a genuine unitary representation of the same dimension. We present a complete classification of uniformly HS stable groups among finitely generated residually finite ones. Necessity of the residual finiteness assumption is discussed. A similar result is shown to hold assuming only amenability.