In this work, we present a framework for the matrix-free solution to a monolithic quasi-static phase-field fracture model with geometric multigrid methods. Using a standard matrix based approach within the Finite Element Method requires lots of memory, which eventually becomes a serious bottleneck. A matrix-free approach overcomes this problems and greatly reduces the amount of required memory, allowing to solve larger problems on available hardware. One key challenge is concerned with the crack irreversibility for which a primal-dual active set method is employed. Here, the Active-Set values of fine meshes must be available on coarser levels of the multigrid algorithm. The developed multigrid method provides a preconditioner for a generalized minimal residual solver. This method is used for solving the linear equations inside Newton's method for treating the overall nonlinear-monolithic discrete displacement/phase-field formulation. Several numerical examples demonstrate the performance and robustness of our solution technology. Mesh refinement studies, variations in the phase-field regularization parameter, iterations numbers of the linear and nonlinear solvers, and some parallel performances are conducted to substantiate the efficiency of the proposed solver for single fractures, multiple pressurized fractures, and a L-shaped panel test in three dimensions.