The difficulties in dealing with discontinuities related to a sharp crack are overcome in the phase-field approach for fracture by modeling the crack as a diffusive object being described by a continuous field having high gradients. The discrete crack limit case is approached for a small length-scale parameter that controls the width of the transition region between the fully broken and the undamaged phases. From a computational standpoint, this necessitates fine meshes, at least locally, in order to accurately resolve the phase-field profile. In the classical approach, phase-field models are computed on a fixed mesh that is a priori refined in the areas where the crack is expected to propagate. This on the other hand curbs the convenience of using phase-field models for unknown crack paths and its ability to handle complex crack propagation patterns. In this work, we overcome this issue by employing the multi-level hp-refinement technique that enables a dynamically changing mesh which in turn allows the refinement to remain local at singularities and high gradients without problems of hanging nodes. Yet, in case of complex geometries, mesh generation and in particular local refinement becomes non-trivial. We address this issue by integrating a two-dimensional phase-field framework for brittle fracture with the finite cell method (FCM). The FCM based on high-order finite elements is a non-geometry-conforming discretization technique wherein the physical domain is embedded into a larger fictitious domain of simple geometry that can be easily discretized. This facilitates mesh generation for complex geometries and supports local refinement. Numerical examples including a comparison to a validation experiment illustrate the applicability of the multi-level hp-refinement and the FCM in the context of phase-field simulations.