It has been known for a while that there is spontaneous breaking of Lorentz symmetry in the nonzero charged sectors of quantum electrodynamics due to the infrared problem of soft photons. More recently, it has also been suggested that similar results hold for color transformations in a non-Abelian gauge theory. Here, we show that an action where a diffeomorphism has been carried out for the part describing hard gauge particles and matter fields can be used to analyze these issues. In addition to rederiving old results in this formalism, we also show that color transformations cannot be unitarily implemented on perturbative gluon states if gluon fields of arbitrarily low energy are allowed. Implications for confinement and mass gap are briefly commented upon.