Topological property of a t2g 5 system with a honeycomb lattice structure

A t₂^{g 5} system with a honeycomb lattice structure such as Na₂IrO₃ was firstly proposed as a topological insulator even though Na₂IrO₃ and its isostructural materials in nature have been turned out to be a Mott insulator with magnetic order. Here we theoretically revisit the topological property based on a minimal tight-binding Hamiltonian for three t_{2 g} bands incorporating a strong spin orbit coupling and two types of the first nearest-neighbor (NN) hopping channel between transition metal ions, i.e., the hopping (t₁) mediated by edge-shared ligands and the direct hopping (t₁^') between t_{2 g} orbitals via d d σ bonding. We demonstrate that the topological phase transition takes place by varying only these hopping parameters with the relative strength parametrized by θ , i.e., t₁=t cosθ and t₁^'=t sinθ . We also explore the effect of the second and third NN hopping channels, and the trigonal distortion on the topological phase for the whole range of θ . Furthermore, we examine the electronic and topological phases in the presence of on-site Coulomb repulsion U . Employing the cluster perturbation theory, we show that, with increasing U , a trivial or topological band insulator in the absence of U can be transferred into a Mott insulator with nontrivial or trivial band topology. We also show that the main effect of the Hund's coupling can be understood simply as the renormalization of U . We briefly discuss the relevance of our results to the existing materials.