A $P_4$-free graph is called a cograph. In this paper we partially characterize finite groups whose power graph is a cograph. As we will see, this problem is a generalization of the determination of groups in which every element has prime power order, first raised by Graham Higman in 1957 and fully solved very recently. First we determine all groups $G$ and $H$ for which the power graph of $G\times H$ is a cograph. We show that groups whose power graph is a cograph can be characterised by a condition only involving elements whose orders are prime or the product of two (possibly equal) primes. Some important graph classes are also taken under consideration. For finite simple groups we show that in most of the cases their power graphs are not cographs: the only ones for which the power graphs are cographs are certain groups PSL$(2,q)$ and Sz$(q)$ and the group PSL$(3,4)$. However, a complete determination of these groups involves some hard number-theoretic problems.