Topological semimetal-to-insulator phase transition between noncollinear and noncoplanar multiple-Q states on a square-to-triangular lattice

Noncollinear and noncoplanar magnetic orders lead to unusual electronic structures and transport properties. We investigate two types of multiple-Q magnetically ordered states and a topological phase transition between them in two dimensions. One is a coplanar but noncollinear double-Q state on a square lattice, which is a semimetal accommodating massless Dirac electrons. The other is a noncoplanar triple-Q state on a triangular lattice, which is a Chern insulator showing the quantum anomalous Hall effect. We discuss the peculiar electronic structures in these two multiple-Q states in a unified way on the basis of the Kondo lattice model, which suggests a quantum phase transition between the two states in a continuous change of lattice geometry between the square and triangular lattices. We systematically examine the possibility of such a transition by using the mean-field approximation for the ground state of the periodic Anderson model. After clarifying the parameter region in which the double-Q (triple-Q ) state is stabilized on the square (triangular) lattice, we show that a continuous topological phase transition indeed takes place between the double-Q Dirac semimetal and the triple-Q Chern insulator on the square-to-triangular lattice. The nature of the transition is discussed by the topologically protected edge states as well as the bulk magnetic and electronic properties. The results indicate that unusual critical phenomena may occur at finite temperature related with multiple-Q chiral spin-liquid states.