Competing structures in two dimensions: Square-to-hexagonal transition

We study a system of particles in two dimensions interacting via a dipolar long-range potential D /r³ and subject to a square-lattice substrate potential V (r ) with amplitude V and lattice constant b . The isotropic interaction favors a hexagonal arrangement of the particles with lattice constant a , which competes against the square symmetry of the underlying substrate lattice. We determine the minimal-energy states at fixed external pressure p generating the commensurate density n =1 /b²=(4^{/3 ) 1 /2}/a² in the absence of thermal and quantum fluctuations, using both analytical techniques based on the harmonic and continuum elastic approximations as well as numerical relaxation of particle configurations. At large substrate amplitude V >0.2 e_D, with e_D=D /b³ the dipolar energy scale, the particles reside in the substrate minima and hence arrange in a square lattice. Upon decreasing V , the square lattice turns unstable with respect to a zone-boundary shear mode and deforms into a period-doubled zigzag lattice. Analytic and numerical results show that this period-doubled phase in turn becomes unstable at V ≈0.074 e_D towards a nonuniform phase developing an array of domain walls or solitons; as the density of solitons increases, the particle arrangement approaches that of a rhombic (or isosceles triangular) lattice. At a yet smaller substrate value estimated as V ≈0.046 e_D, a further solitonic transition establishes a second nonuniform phase which smoothly approaches the hexagonal (or equilateral triangular) lattice phase with vanishing amplitude V . At small but finite amplitude V , the hexagonal phase is distorted and hexatically locked at an angle of φ ≈3 .8^∘ with respect to the substrate lattice. The square-to-hexagonal transformation in this two-dimensional commensurate-incommensurate system thus involves a complex pathway with various nontrivial lattice- and modulated phases.