We propose the ZQ Berry phase as a topological invariant for higher-order symmetry-protected topological (HOSPT) phases for two- and three-dimensional systems. It is topologically stable for electron-electron interactions assuming the gap remains open. As a concrete example, we show that the Berry phase is quantized in Z4 and characterizes the HOSPT phase of the extended Benalcazar-Bernevig-Hughes (BBH) model, which contains the next-nearest-neighbor hopping and the intersite Coulomb interactions. In addition, we introduce the Z4 Berry phase for the spin-model analog of the BBH model. Furthermore, we demonstrate the Berry phase is quantized in Z4 for the three-dimensional version of the BBH model. We also confirm the bulk-corner correspondence between the Z4 Berry phase and the corner states in the HOSPT phases.