Destruction of the long-range order in a vortex crystal by a random potential is expected to be accompanied by the formation of a glasslike phase-the so-called vortex glass. The properties of such a phase are investigated in the framework of the self-consistent harmonic approximation, taking into account the possibility of replica symmetry breaking. The main attention is given to the problem of the uniaxial vortex glass in which the vortices are free to move only in one direction. We obtain the result that in the two-dimensional case, upon lowering the temperature, a phase transition takes place between the phase in which the random potential is irrelevant, to the phase with one-step replica symmetry breaking. For 2<D<4 the random potential is always relevant and the replica symmetry breaking is of the hierarchical type. In both cases, the fluctuations of the displacement in the glassy phase diverge logarithmically. The same conclusions are shown to be valid for the case of a biaxial vortex glass in the absence of dislocations. The results obtained are also applicable to the description of the pinning of charge density waves.