Ground-state competition of two-component bosons in an optical lattice near a Feshbach resonance

We investigate the ground-state properties of an equal mixture of two species of bosons in its Mott-insulator phase at a filling factor of two per site. We identify one type of spin triplet-singlet transition through the competition of the ground state. When the on-site interaction is weak (U<U_c) the two particles prefer to stay in the lowest band and with weak tunneling between neighboring sites the system is mapped into an effective spin-1 ferromagnetic exchange Hamiltonian. When the interaction is tuned by a Feshbach resonance to be large enough (U>U_c) , the higher band will be populated. Due to the orbital coupling term S⁺S^- in the Hamiltonian, the two atoms in different orbits on a site would form an on-site singlet. For a non-SU(2)-symmetric model, easy-axis or easy-plane ferromagnetic spin exchange models may be realized, corresponding to phase separation or counterflow superfluidity, respectively.