Crystal structures of bromine under high pressure have been studied by employing plane-wave pseudopotential method with the generalized gradient approximation. It is found that the band overlap in the molecular Cmca phase, which causes the pressure-induced insulator-to-metal transition, occurs at about 55GPa . Geometry optimization shows that the bromine changes to a face-centered orthorhombic (fco) phase with equal interatomic distances d1=d2=d3 at about 75GPa , but this fco structure is mechanically unstable with shear elastic stiffness coefficient C66<0 . For understanding the structure of this phase, we have modeled an incommensurate structure by a rational approximation with modulation vector k=(0.25,0,0) according to the previous research results in solid iodine. Our results show that the enthalpy of this modulated phase is lower than that of the fco solid, and the elastic stiffness coefficients (Cij) satisfy the Born stability criteria, indicating that the modulated structure is more thermodynamically stable and mechanically stable. In addition, through comparing the x-ray diffraction patterns of our structure with the experimental one, we conclude that the structure of bromine phase V is close to our modulated structure. It is clearly illustrated that the phase transition from Cmca phase to the incommensurate phase is associated with the instability of the shear elastic stiffness coefficient C44 which is related to the softening of the long-wavelength part of the transverse branch near the center of the first Brillouin zone. With the increasing of pressure, the modulated phase transforms into the monatomic phase II with body-centered orthorhombic structure at about 100GPa , which is in agreement with the experimental result.