Histogram Monte Carlo simulation of the geometrically frustrated XY antiferromagnet with biquadratic exchange

Histogram Monte Carlo simulation is used to investigate effects of biquadratic exchange J₂ on phase transitions of a three-dimensional classical XY antiferromagnet with frustration induced by the antiferromagnetic exchange J₁ and the stacked triangular lattice geometry. The biquadratic exchange is considered negative (antiferroquadrupolar) within the triangular planes and positive (ferroquadrupolar) between the planes. The phase diagram obtained features a variety of interesting phenomena arising from the presence of both the biquadratic exchange and frustration. In a strong biquadratic exchange limit (\|J₁\|/\|J₂\|<=0.25), the antiferroquadrupolar phase transition which is of second order is followed by the antiferromagnetic one which can be either first or second order. The separate antiferroquadrupolar and antiferromagnetic second-order transitions are found to belong to the chiral XY and Ising universality classes, respectively. If the biquadratic exchange is reduced, both transitions are found to be first order and occur simultaneously in a wide region of \|J₁\|/\|J₂\|. However, if \|J₂\|-->0, the transition changes to the second-order one with the chiral universality class critical behavior.