We present an $hr$-adaptivity framework for optimization of high-order meshes. This work extends the $r$-adaptivity method for mesh optimization by Dobrev et al., where we utilized the Target-Matrix Optimization Paradigm (TMOP) to minimize a functional that depends on each element's current and target geometric parameters: element aspect-ratio, size, skew, and orientation. Since fixed mesh topology limits the ability to achieve the target size and aspect-ratio at each position, in this paper we augment the $r$-adaptivity framework with nonconforming adaptive mesh refinement to further reduce the error with respect to the target geometric parameters. The proposed formulation, referred to as $hr$-adaptivity, introduces TMOP-based quality estimators to satisfy the aspect-ratio-target via anisotropic refinements and size-target via isotropic refinements in each element of the mesh. The methodology presented is purely algebraic, extends to both simplices and hexahedra/quadrilaterals of any order, and supports nonconforming isotropic and anisotropic refinements in 2D and 3D. Using a problem with a known exact solution, we demonstrate the effectiveness of $hr$-adaptivity over both $r-$ and $h$-adaptivity in obtaining similar accuracy in the solution with significantly fewer degrees of freedom. We also present several examples that show that $hr$-adaptivity can help satisfy geometric targets even when $r$-adaptivity fails to do so, due to the topology of the initial mesh.